2020-03-11T22:20:29.443 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:20:29.444 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.kp5TS_.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848253 2020-03-11T22:20:29.465 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:20:29.525 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848253 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848253' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.16501 2020-03-11T22:20:29.525 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:20:29.527 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:20:29.527 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:20:29.529 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:20:29.553 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:20:29.553 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:20:29.553 INFO:teuthology.packaging:ref: None 2020-03-11T22:20:29.554 INFO:teuthology.packaging:tag: None 2020-03-11T22:20:29.554 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:20:29.554 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:20:29.554 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:20:29.779 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:20:29.780 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:20:29.789 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:20:29.790 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:20:29.832 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:20:29.832 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:20:29.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:39.903 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848253', 'machine_type': 'smithi'} 2020-03-11T22:20:39.920 DEBUG:teuthology.lock.ops:locked ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:20:39.949 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:20:40.151 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 8.1 2020-03-11T22:20:40.496 INFO:teuthology.orchestra.console:Power off smithi045 2020-03-11T22:20:40.496 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:20:40.520 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:44.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:48.658 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:48.868 INFO:teuthology.orchestra.console:Power off for smithi045 completed 2020-03-11T22:20:48.969 INFO:teuthology.orchestra.console:Power on smithi045 2020-03-11T22:20:48.969 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:20:48.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:53.012 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:57.125 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:57.334 INFO:teuthology.orchestra.console:Power on for smithi045 completed 2020-03-11T22:20:57.435 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish 2020-03-11T22:23:28.696 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:37.755 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:46.810 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:55.866 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:04.922 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:13.978 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:23.034 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:54.787 INFO:teuthology.orchestra.run.smithi045:> hostname 2020-03-11T22:24:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:smithi136 2020-03-11T22:24:55.113 INFO:teuthology.orchestra.run.smithi045:> grep smithi136 /etc/hosts 2020-03-11T22:24:55.167 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136 2020-03-11T22:24:55.167 INFO:teuthology.orchestra.run.smithi045:> sudo hostname smithi045 2020-03-11T22:24:55.288 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/smithi136/smithi045/g' /etc/hosts 2020-03-11T22:24:55.320 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/172.21.15.136/172.21.15.45/g' /etc/hosts 2020-03-11T22:24:55.391 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/smithi136/smithi045/g' /etc/hostname 2020-03-11T22:24:55.462 INFO:teuthology.orchestra.run.smithi045:> sudo hostname smithi045 2020-03-11T22:24:55.533 INFO:teuthology.provision.fog.smithi045:Deploy complete! 2020-03-11T22:24:55.535 DEBUG:teuthology.parallel:result is None 2020-03-11T22:24:55.636 INFO:teuthology.lock.ops:Checking smithi045.front.sepia.ceph.com 2020-03-11T22:24:55.637 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:24:55.656 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:24:55.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:24:55.657 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:55.726 INFO:teuthology.orchestra.run.smithi045:> true 2020-03-11T22:24:55.916 INFO:teuthology.orchestra.run.smithi045:> uname -m 2020-03-11T22:24:55.970 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2020-03-11T22:24:55.970 INFO:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2020-03-11T22:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Linux" 2020-03-11T22:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8 (Core)" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2020-03-11T22:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:24:56.027 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2020-03-11T22:24:56.039 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:24:56.054 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwLeaAFweiGC8h7DBs5k2uDfJfY4pEHHb6ZKx9s63nO9tmzASJXeX+ZqYvOeQYpKjZtEuDQIhjiuMHHql1YR6dPbHgNtLmJ1IyCwDZXJav7gwyiO1c7VfZRRsQ7UMHuFrRVNu9X71Fz0HomghHs3pA1RY9CTRJbdFKyPTpOwR2gAqZ8KYAe/CqRbiPNrXgDzT+9597bsA5arjEuaXcsN/VNsToJOlSKO9dwkD1DLERni3gMC9f2c0TUkGu6piUNbpULXCV71BOgGgkf/0UM8LxmaZNRnGUoE1ta03Lve9f+GSDpY7ZhgsJ4wmIXPrZltOTl9DrNwYnepunBcGoDGDpykDzjD3w7L0mmamzr2lWHt+nMzZEqSpV9AozEjZf+6IPyNU+RJq8WGKVpwtK2SGeEHG6urg1BRocWvXVMf4qzMbFLkfZzpPJZG7zLgSdG7prwe+D1OXN0N6HU5WcoKQmkZgkYidqs/4f6yyyTjhTWIeQ6kASArszDzG+oCOPR8= 2020-03-11T22:24:56.054 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:24:56.090 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:24:56.096 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:24:56.106 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:24:56.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:24:56.135 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:20:39.910859', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi045.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEA3Ktt2KJ5WhgSZRRhkF1/21MbXnYEk3mPcLrTe5iq4MXsFA3AHFLd4rWn+uGBsJB8+aoU5QNt3qTPsG4LlPe0=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848253'} 2020-03-11T22:24:56.135 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:24:56.145 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:24:56.145 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:24:56.193 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:24:56.194 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:24:56.203 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:24:56.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:24:56.205 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:56.282 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:24:56.292 INFO:teuthology.orchestra.run.smithi045:> uname -m 2020-03-11T22:24:56.468 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2020-03-11T22:24:56.468 INFO:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2020-03-11T22:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Linux" 2020-03-11T22:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8 (Core)" 2020-03-11T22:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:24:56.525 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2020-03-11T22:24:56.548 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:24:56.572 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:24:56.581 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:24:56.582 INFO:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:24:56.595 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:24:56.604 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:24:56.604 INFO:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:24:56.652 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:24:56.681 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:24:56.701 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:24:56.701 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:24:56.701 INFO:teuthology.orchestra.run.smithi045:> uname -r 2020-03-11T22:24:56.716 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:24:56.716 INFO:teuthology.task.kernel:Running kernel on smithi045: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:24:56.716 INFO:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2020-03-11T22:24:59.253 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 1 day, 6:59:05 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:24:59.790 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:24:59.790 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:24:59.983 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:24:59.984 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:24:59.984 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:25:00.107 INFO:teuthology.orchestra.run.smithi045:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:25:00.538 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 1 day, 6:59:07 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:00.960 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:25:00.983 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:00.983 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:00.983 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalling: 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2020-03-11T22:25:00.985 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 0 2020-03-11T22:25:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:Operation aborted. 2020-03-11T22:25:01.075 INFO:teuthology.orchestra.run.smithi045.stdout:Is this ok [y/N]: 2020-03-11T22:25:01.075 INFO:teuthology.orchestra.run.smithi045:> sudo yum reinstall -y kernel || true 2020-03-11T22:25:01.501 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 1 day, 6:59:08 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:01.880 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:25:01.881 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:01.881 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:01.881 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalling: 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:25:01.882 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:01.883 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2020-03-11T22:25:01.883 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 0 2020-03-11T22:25:01.883 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:25:01.982 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 17 MB/s | 1.5 MB 00:00 2020-03-11T22:25:01.995 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2020-03-11T22:25:01.996 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:25:01.996 INFO:teuthology.orchestra.run.smithi045.stdout:Total 25 MB/s | 3.1 MB 00:00 2020-03-11T22:25:02.022 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:25:02.208 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:25:02.208 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:25:02.290 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:25:02.292 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:25:02.633 INFO:teuthology.orchestra.run.smithi045.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:25:02.815 INFO:teuthology.orchestra.run.smithi045.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:02.815 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:02.926 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:02.927 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:03.203 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:03.203 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:25:03.203 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:03.203 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:03.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:25:03.479 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:03.479 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalled: 2020-03-11T22:25:03.479 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:25:03.479 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:03.480 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:25:03.600 INFO:teuthology.orchestra.run.smithi045:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:03.663 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:02 PM UTC 2020-03-11T22:25:03.665 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:03.665 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:25:03.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:25:03.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:25:03.665 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:25:03.665 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:25:03.666 INFO:teuthology.orchestra.run.smithi045:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:03.709 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:02 PM UTC 2020-03-11T22:25:03.710 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:03.711 INFO:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2020-03-11T22:25:03.748 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2020-03-11T22:25:03.748 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2020-03-11T22:25:03.748 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.02 2020-03-11T22:25:03.748 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 78.el8 2020-03-11T22:25:03.748 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2020-03-11T22:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:Group : System Environment/Base 2020-03-11T22:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 9192691 2020-03-11T22:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2020-03-11T22:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Relocations : (not relocatable) 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : CentOS BuildSystem 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2020-03-11T22:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2020-03-11T22:25:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:25:03.752 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:25:03.752 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:25:03.753 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:25:03.753 INFO:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:25:06.846 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2020-03-11T22:25:13.418 INFO:teuthology.orchestra.run.smithi045.stderr:done 2020-03-11T22:25:13.419 INFO:teuthology.orchestra.run.smithi045:> mktemp 2020-03-11T22:25:13.446 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.Yyf7Ko4Gya 2020-03-11T22:25:13.446 INFO:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Yyf7Ko4Gya 2020-03-11T22:25:13.535 INFO:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.Yyf7Ko4Gya 2020-03-11T22:25:13.648 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.Yyf7Ko4Gya is 4KB 2020-03-11T22:25:13.696 INFO:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.Yyf7Ko4Gya 2020-03-11T22:25:13.711 INFO:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:25:13.789 INFO:teuthology.orchestra.run.smithi045.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:25:13.789 INFO:teuthology.orchestra.run.smithi045.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:25:13.790 INFO:teuthology.orchestra.run.smithi045.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:25:13.791 INFO:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:25:13.947 INFO:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2020-03-11T22:25:43.975 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:25:43.975 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:25:43.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:25:43.976 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:02.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2020-03-11T22:26:02.342 DEBUG:teuthology.misc:waited 18.3669250011 2020-03-11T22:26:03.343 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:26:03.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:03.343 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:06.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2020-03-11T22:26:06.406 DEBUG:teuthology.misc:waited 22.4309308529 2020-03-11T22:26:07.406 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:26:07.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:07.407 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:09.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2020-03-11T22:26:09.482 DEBUG:teuthology.misc:waited 25.5069179535 2020-03-11T22:26:10.482 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:26:10.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:10.483 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:12.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2020-03-11T22:26:12.550 DEBUG:teuthology.misc:waited 28.574947834 2020-03-11T22:26:13.551 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:26:13.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:13.551 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:15.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2020-03-11T22:26:15.626 DEBUG:teuthology.misc:waited 31.6511290073 2020-03-11T22:26:16.626 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:26:16.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:16.627 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:16.629 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:27:16.629 DEBUG:teuthology.misc:waited 92.6541569233 2020-03-11T22:27:17.629 INFO:teuthology.misc:trying to connect to ubuntu@smithi045.front.sepia.ceph.com 2020-03-11T22:27:17.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:17.630 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:18.007 INFO:teuthology.orchestra.run.smithi045:> true 2020-03-11T22:27:18.307 DEBUG:teuthology.misc:waited 94.3318619728 2020-03-11T22:27:19.307 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:27:19.308 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:27:19.308 INFO:teuthology.orchestra.run.smithi045:> uname -r 2020-03-11T22:27:19.328 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:27:19.328 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:27:19.328 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:27:20.329 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:27:20.329 INFO:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:27:20.442 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2020-03-11T22:27:20.454 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:27:20.988 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:27:20.988 INFO:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:27:21.006 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:27:21.015 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:27:21.026 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:27:21.027 INFO:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:27:21.074 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:27:21.083 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:27:21.084 INFO:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:27:21.153 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:21.171 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:21.173 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:27:21.184 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:27:21.185 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:27:21.247 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:27:21.260 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:27:21.260 INFO:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:27:21.304 INFO:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:21.379 INFO:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:21.455 INFO:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:21.488 INFO:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:21.544 INFO:teuthology.orchestra.run.smithi045:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:27:21.630 INFO:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2020-03-11T22:27:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:27:22.146 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:27:22.157 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:27:22.157 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:27:22.239 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:27:22.254 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:27:22.255 INFO:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2020-03-11T22:27:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: [ OK ] 2020-03-11T22:27:22.326 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:27:22.327 INFO:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2020-03-11T22:27:22.346 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2020-03-11T22:27:22.347 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:27:22.347 INFO:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:27:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:27:22.418 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:27:22.419 INFO:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:27:22.489 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:27:22.505 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:27:22.505 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:27:22.706 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:27:22.713 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:27:23.688 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:27:26.018 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:27:29.360 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:27:29.412 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.175) 0:00:00.175 ******* 2020-03-11T22:27:32.268 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi045.front.sepia.ceph.com] 2020-03-11T22:27:32.369 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:32 +0000 (0:00:02.956) 0:00:03.132 ******* 2020-03-11T22:27:32.766 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:27:32.768 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:27:32.830 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:32 +0000 (0:00:00.461) 0:00:03.593 ******* 2020-03-11T22:27:33.436 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:27:33.437 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:27:33.479 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:33 +0000 (0:00:00.648) 0:00:04.242 ******* 2020-03-11T22:27:33.505 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:27:33.507 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:33.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:33 +0000 (0:00:00.063) 0:00:04.305 ******* 2020-03-11T22:27:33.929 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:33.931 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:33.976 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:33 +0000 (0:00:00.433) 0:00:04.739 ******* 2020-03-11T22:27:35.031 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:27:35.031 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965654.0-278232978648254/source", "state": "file", "uid": 0} 2020-03-11T22:27:35.073 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:35 +0000 (0:00:01.097) 0:00:05.836 ******* 2020-03-11T22:27:35.532 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:27:35.533 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:27:35.534 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:27:35.630 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:27:35.638 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:27:35.861 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:35 +0000 (0:00:00.787) 0:00:06.624 ******* 2020-03-11T22:27:36.545 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-03-11T22:27:36.546 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2020-03-11T22:27:36.635 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:27:36.636 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:36 +0000 (0:00:00.773) 0:00:07.398 ******* 2020-03-11T22:27:37.446 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:27:37.447 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.433060", "end": "2020-03-11 22:27:37.421501", "rc": 0, "start": "2020-03-11 22:27:36.988441", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:37.484 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.849) 0:00:08.247 ******* 2020-03-11T22:27:37.705 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:27:37.706 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.007697", "end": "2020-03-11 22:27:37.677589", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:27:37.669892", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:37.753 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.268) 0:00:08.516 ******* 2020-03-11T22:27:37.789 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:27:37.791 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:27:37.829 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.076) 0:00:08.592 ******* 2020-03-11T22:27:37.854 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:27:37.856 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:37.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.062) 0:00:08.655 ******* 2020-03-11T22:27:37.918 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:27:37.919 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:37.954 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.062) 0:00:08.717 ******* 2020-03-11T22:27:37.981 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:27:37.983 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.027 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.072) 0:00:08.790 ******* 2020-03-11T22:27:38.054 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:27:38.056 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.091 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.063) 0:00:08.854 ******* 2020-03-11T22:27:38.117 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:27:38.118 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.158 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.067) 0:00:08.921 ******* 2020-03-11T22:27:38.205 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:27:38.206 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.208 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.219 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.220 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.231 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.232 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.242 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.243 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.254 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.265 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.266 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.278 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.288 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.299 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.299 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.301 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.311 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.312 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.323 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.324 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.335 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.336 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.347 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.347 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.358 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.359 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.370 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.370 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.381 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.392 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.393 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.404 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.406 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.416 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.417 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.428 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.428 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.440 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.440 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.452 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.452 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.463 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.474 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.475 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.485 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.486 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.497 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.508 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.509 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.520 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.530 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.531 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.542 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.542 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.553 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.564 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.566 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.577 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.577 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.588 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.590 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.600 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.601 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.623 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.623 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.647 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.647 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.659 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.659 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.671 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.672 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.672 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:38.684 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.684 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.696 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.718 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.719 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.730 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.731 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.742 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.742 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.753 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.754 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.766 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.766 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.778 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.778 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.789 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.800 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.801 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.812 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.812 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.823 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.834 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.835 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.846 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.846 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.857 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.868 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.869 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.880 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.890 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.891 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.902 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.904 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.916 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.916 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.927 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.928 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.939 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.951 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.951 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.007 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.848) 0:00:09.770 ******* 2020-03-11T22:27:39.033 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:27:39.072 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.065) 0:00:09.835 ******* 2020-03-11T22:27:39.098 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:27:39.100 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.145 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.072) 0:00:09.908 ******* 2020-03-11T22:27:39.172 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:27:39.173 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.211 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.066) 0:00:09.974 ******* 2020-03-11T22:27:39.247 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:27:39.248 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.286 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.074) 0:00:10.049 ******* 2020-03-11T22:27:39.311 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:27:39.313 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.352 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.066) 0:00:10.115 ******* 2020-03-11T22:27:39.379 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:27:39.380 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.065) 0:00:10.181 ******* 2020-03-11T22:27:39.454 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:27:39.455 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.466 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.467 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.487 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.489 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.499 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.510 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.510 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.521 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.534 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.534 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.548 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.548 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.560 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.560 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.571 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.572 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.583 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.583 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.595 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.595 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.609 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.610 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.621 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.622 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.633 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.643 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.654 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.655 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.666 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.667 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.679 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.680 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.691 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.692 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.704 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.704 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.715 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.715 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.726 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.727 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.737 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:39.782 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:00.363) 0:00:10.544 ******* 2020-03-11T22:27:40.684 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:27:40.684 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:27:40.728 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:00.946) 0:00:11.491 ******* 2020-03-11T22:27:40.764 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:27:40.765 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => { "msg": "Host smithi045.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:27:40.805 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:00.076) 0:00:11.568 ******* 2020-03-11T22:27:40.852 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:27:40.854 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:27:40.890 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:00.085) 0:00:11.653 ******* 2020-03-11T22:27:40.916 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:27:40.917 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:40.953 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:00.062) 0:00:11.716 ******* 2020-03-11T22:27:41.288 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:27:41.289 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.125864", "end": "2020-03-11 22:27:41.260275", "rc": 0, "start": "2020-03-11 22:27:41.134411", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:27:41.337 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.383) 0:00:12.100 ******* 2020-03-11T22:27:41.365 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:27:41.366 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.404 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.067) 0:00:12.167 ******* 2020-03-11T22:27:41.431 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:27:41.432 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.065) 0:00:12.233 ******* 2020-03-11T22:27:41.506 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:27:41.508 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.544 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.073) 0:00:12.307 ******* 2020-03-11T22:27:41.580 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:27:41.581 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.617 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.073) 0:00:12.380 ******* 2020-03-11T22:27:41.644 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:41.645 INFO:teuthology.task.ansible.out:TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.692 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.074) 0:00:12.455 ******* 2020-03-11T22:27:41.729 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:27:41.730 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.771 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.079) 0:00:12.534 ******* 2020-03-11T22:27:41.819 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:27:41.820 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:41.821 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:41.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.092) 0:00:12.627 ******* 2020-03-11T22:27:41.900 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:27:41.901 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:41.943 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.079) 0:00:12.706 ******* 2020-03-11T22:27:41.970 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:27:41.971 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.028 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.084) 0:00:12.791 ******* 2020-03-11T22:27:42.065 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:27:42.066 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:42.101 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.073) 0:00:12.864 ******* 2020-03-11T22:27:42.137 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:27:42.139 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.176 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.074) 0:00:12.939 ******* 2020-03-11T22:27:42.202 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:27:42.203 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:42.240 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.063) 0:00:13.003 ******* 2020-03-11T22:27:42.265 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:27:42.267 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.305 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.064) 0:00:13.067 ******* 2020-03-11T22:27:42.341 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:27:42.342 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.378 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.073) 0:00:13.141 ******* 2020-03-11T22:27:42.404 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:27:42.405 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.442 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.064) 0:00:13.205 ******* 2020-03-11T22:27:42.468 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:27:42.470 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.510 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.067) 0:00:13.273 ******* 2020-03-11T22:27:42.537 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:27:42.538 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:42.575 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.064) 0:00:13.338 ******* 2020-03-11T22:27:42.602 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:27:42.604 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.648 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.073) 0:00:13.411 ******* 2020-03-11T22:27:42.686 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:27:42.687 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.725 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.076) 0:00:13.488 ******* 2020-03-11T22:27:42.753 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:27:42.754 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.792 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.067) 0:00:13.555 ******* 2020-03-11T22:27:42.830 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:27:42.831 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.074) 0:00:13.630 ******* 2020-03-11T22:27:42.905 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:27:42.906 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.940 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.073) 0:00:13.703 ******* 2020-03-11T22:27:42.967 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-11T22:27:42.968 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.005 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.064) 0:00:13.768 ******* 2020-03-11T22:27:43.032 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-11T22:27:43.034 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:43.070 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.064) 0:00:13.833 ******* 2020-03-11T22:27:43.099 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:27:43.137 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.066) 0:00:13.900 ******* 2020-03-11T22:27:43.175 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:27:43.176 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.215 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.078) 0:00:13.978 ******* 2020-03-11T22:27:43.482 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:27:43.483 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:43.524 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.309) 0:00:14.287 ******* 2020-03-11T22:27:44.058 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:27:44.058 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:27:44.554 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:27:44.598 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:44 +0000 (0:00:01.073) 0:00:15.361 ******* 2020-03-11T22:27:45.126 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:27:45.126 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:27:45.607 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:27:45.651 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:01.052) 0:00:16.413 ******* 2020-03-11T22:27:45.688 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.733 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.082) 0:00:16.496 ******* 2020-03-11T22:27:45.761 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:27:45.762 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.810 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.076) 0:00:16.573 ******* 2020-03-11T22:27:45.846 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:27:45.849 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.886 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.076) 0:00:16.649 ******* 2020-03-11T22:27:45.925 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:27:45.927 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.970 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.083) 0:00:16.733 ******* 2020-03-11T22:27:45.998 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:27:46.000 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.037 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.067) 0:00:16.800 ******* 2020-03-11T22:27:46.064 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:27:46.066 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.109 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.071) 0:00:16.872 ******* 2020-03-11T22:27:46.135 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.176 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.065) 0:00:16.937 ******* 2020-03-11T22:27:46.201 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:27:46.203 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.238 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.064) 0:00:17.001 ******* 2020-03-11T22:27:46.264 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:27:46.265 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.306 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.068) 0:00:17.069 ******* 2020-03-11T22:27:47.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:27:47.523 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.007380", "end": "2020-03-11 22:27:47.500332", "rc": 0, "start": "2020-03-11 22:27:46.492952", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:47.564 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:01.257) 0:00:18.327 ******* 2020-03-11T22:27:50.790 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:27:50.792 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:27:50.843 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:03.279) 0:00:21.606 ******* 2020-03-11T22:27:50.870 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:27:50.872 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.914 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.066) 0:00:21.673 ******* 2020-03-11T22:27:50.938 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.974 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.063) 0:00:21.737 ******* 2020-03-11T22:27:51.029 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:27:51.030 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:51.072 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:51 +0000 (0:00:00.097) 0:00:21.835 ******* 2020-03-11T22:27:51.109 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:27:51.110 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:51.148 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:51 +0000 (0:00:00.076) 0:00:21.911 ******* 2020-03-11T22:27:51.673 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:27:51.674 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:27:51.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:51 +0000 (0:00:00.570) 0:00:22.482 ******* 2020-03-11T22:27:52.689 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:27:52.691 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:27:52.738 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:52 +0000 (0:00:01.018) 0:00:23.501 ******* 2020-03-11T22:27:53.288 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:27:53.289 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:27:53.802 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:27:54.274 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:27:54.321 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:01.582) 0:00:25.084 ******* 2020-03-11T22:27:54.533 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:27:54.535 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:27:54.573 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:00.252) 0:00:25.336 ******* 2020-03-11T22:27:55.069 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:27:55.071 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:27:55.550 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:27:56.024 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:27:56.064 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:01.490) 0:00:26.827 ******* 2020-03-11T22:27:56.090 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:27:56.092 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:56.128 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:00.063) 0:00:26.890 ******* 2020-03-11T22:27:56.164 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:27:56.166 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:27:56.210 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:00.082) 0:00:26.973 ******* 2020-03-11T22:27:56.433 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:27:56.435 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:27:56.476 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:00.265) 0:00:27.238 ******* 2020-03-11T22:27:56.512 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:27:56.514 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:56.524 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:56.535 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:56.546 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:56.585 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:00.109) 0:00:27.348 ******* 2020-03-11T22:27:57.577 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:27:57.578 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:27:57.618 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:57 +0000 (0:00:01.032) 0:00:28.381 ******* 2020-03-11T22:27:57.643 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:27:57.645 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:57.687 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:57 +0000 (0:00:00.068) 0:00:28.450 ******* 2020-03-11T22:27:58.212 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:27:58.213 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:27:58.252 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:00.565) 0:00:29.015 ******* 2020-03-11T22:27:58.780 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:27:58.781 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:27:58.822 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:00.569) 0:00:29.585 ******* 2020-03-11T22:27:58.859 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:27:58.860 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:58.898 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:00.076) 0:00:29.661 ******* 2020-03-11T22:27:59.128 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:27:59.128 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:59.166 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.267) 0:00:29.929 ******* 2020-03-11T22:27:59.432 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:27:59.434 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:27:59.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.303) 0:00:30.233 ******* 2020-03-11T22:27:59.499 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:59.544 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.074) 0:00:30.307 ******* 2020-03-11T22:28:00.056 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:28:00.058 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:28:00.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.568) 0:00:30.876 ******* 2020-03-11T22:28:01.039 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:28:01.041 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:26:47 UTC", "ActiveEnterTimestampMonotonic": "14695348", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target tmp.mount nss-lookup.target sysinit.target systemd-tmpfiles-setup.service systemd-journald.socket basic.target system.slice -.mount var-run.mount time-sync.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:26:47 UTC", "AssertTimestampMonotonic": "14694468", "Before": "shutdown.target multi-user.target plymouth-quit.service xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:26:47 UTC", "ConditionTimestampMonotonic": "14694468", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:28:01.042 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2381", "ExecMainStartTimestamp": "Wed 2020-03-11 22:26:47 UTC", "ExecMainStartTimestampMonotonic": "14695312", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:26:47 UTC] ; stop_time=[n/a] ; pid=2381 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:26:47 UTC", "InactiveExitTimestampMonotonic": "14695348", "InvocationID": "e3413aecf671486cb804ee8eb3d9cac1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:28:01.044 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", "LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2381", "MemoryAccounting": "yes", "MemoryCurrent": "1835008", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:28:01.045 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:26:47 UTC", "StateChangeTimestampMonotonic": "14695348", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:26:47 UTC", "WatchdogTimestampMonotonic": "14695347", "WatchdogUSec": "0"}} 2020-03-11T22:28:01.078 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.965) 0:00:31.841 ******* 2020-03-11T22:28:01.291 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:28:01.293 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001778", "end": "2020-03-11 22:28:01.269457", "rc": 0, "start": "2020-03-11 22:28:01.267679", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:28:01.330 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.251) 0:00:32.093 ******* 2020-03-11T22:28:02.297 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:28:02.299 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:28:02.336 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:01.006) 0:00:33.099 ******* 2020-03-11T22:28:03.308 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:28:03.309 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:28:03.344 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:01.008) 0:00:34.107 ******* 2020-03-11T22:28:04.722 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-11T22:28:04.724 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:28:04.765 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:01.420) 0:00:35.528 ******* 2020-03-11T22:28:22.010 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:28:22.011 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:17.013656", "end": "2020-03-11 22:28:21.973919", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:28:04.960263", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:28:22.049 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:22 +0000 (0:00:17.283) 0:00:52.812 ******* 2020-03-11T22:28:22.562 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:28:22.563 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:28:22.600 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:22 +0000 (0:00:00.551) 0:00:53.363 ******* 2020-03-11T22:28:22.843 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:28:22.844 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.019410", "end": "2020-03-11 22:28:22.814927", "rc": 0, "start": "2020-03-11 22:28:22.795517", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:22.884 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:22 +0000 (0:00:00.283) 0:00:53.647 ******* 2020-03-11T22:28:23.101 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:28:23.103 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002614", "end": "2020-03-11 22:28:23.071577", "rc": 0, "start": "2020-03-11 22:28:23.068963", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:23.144 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:23 +0000 (0:00:00.259) 0:00:53.907 ******* 2020-03-11T22:28:40.096 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:28:40.097 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.743269", "end": "2020-03-11 22:28:40.073168", "rc": 0, "start": "2020-03-11 22:28:23.329899", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:40.135 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:16.990) 0:01:10.898 ******* 2020-03-11T22:28:40.347 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:28:40.349 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:28:40.387 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.252) 0:01:11.150 ******* 2020-03-11T22:28:40.424 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:28:40.425 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:28:40.462 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.074) 0:01:11.225 ******* 2020-03-11T22:28:40.508 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:28:40.510 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:28:40.547 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.084) 0:01:11.310 ******* 2020-03-11T22:28:40.583 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:28:40.584 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:28:40.622 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.074) 0:01:11.385 ******* 2020-03-11T22:28:40.657 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:28:40.659 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:28:40.695 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.073) 0:01:11.458 ******* 2020-03-11T22:28:40.906 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:28:40.907 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:28:40.944 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:40 +0000 (0:00:00.248) 0:01:11.707 ******* 2020-03-11T22:28:41.146 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:28:41.147 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:28:41.185 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:41 +0000 (0:00:00.240) 0:01:11.947 ******* 2020-03-11T22:28:41.210 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:28:41.211 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:41.247 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:41 +0000 (0:00:00.061) 0:01:12.010 ******* 2020-03-11T22:28:41.272 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:28:41.273 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:41.311 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:41 +0000 (0:00:00.064) 0:01:12.074 ******* 2020-03-11T22:28:41.554 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:28:41.555 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:28:41.591 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:41 +0000 (0:00:00.280) 0:01:12.354 ******* 2020-03-11T22:28:42.182 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:28:42.183 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:28:42.220 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:42 +0000 (0:00:00.628) 0:01:12.983 ******* 2020-03-11T22:28:42.247 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:28:42.248 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi045"}, "changed": false} 2020-03-11T22:28:42.285 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:42 +0000 (0:00:00.064) 0:01:13.048 ******* 2020-03-11T22:28:43.421 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:28:43.422 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi045.front.sepia.ceph.com", "ansible_hostname": "smithi045", "ansible_nodename": "smithi045"}, "changed": true, "name": "smithi045"} 2020-03-11T22:28:43.460 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:43 +0000 (0:00:01.175) 0:01:14.223 ******* 2020-03-11T22:28:43.957 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:28:43.958 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:28:44.013 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.552) 0:01:14.776 ******* 2020-03-11T22:28:44.502 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:28:44.502 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:28:44.540 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.526) 0:01:15.303 ******* 2020-03-11T22:28:45.188 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:28:45.188 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:28:45.226 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.686) 0:01:15.989 ******* 2020-03-11T22:28:45.744 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:28:45.783 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.556) 0:01:16.545 ******* 2020-03-11T22:28:46.021 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:28:46.022 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:28:46.059 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:46 +0000 (0:00:00.276) 0:01:16.822 ******* 2020-03-11T22:28:46.332 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:28:46.334 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:28:46.371 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:46 +0000 (0:00:00.311) 0:01:17.134 ******* 2020-03-11T22:28:46.654 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:28:46.656 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:28:46.690 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:46 +0000 (0:00:00.318) 0:01:17.453 ******* 2020-03-11T22:28:47.229 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:28:47.230 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:28:47.268 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:47 +0000 (0:00:00.577) 0:01:18.031 ******* 2020-03-11T22:28:47.780 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:28:47.780 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:28:47.818 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:47 +0000 (0:00:00.549) 0:01:18.580 ******* 2020-03-11T22:28:49.829 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:28:49.831 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.795027", "end": "2020-03-11 22:28:49.803274", "rc": 0, "start": "2020-03-11 22:28:48.008247", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:49.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:49 +0000 (0:00:02.052) 0:01:20.633 ******* 2020-03-11T22:28:50.413 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:28:50.414 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:28:50.891 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:28:51.367 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:28:51.407 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:51 +0000 (0:00:01.536) 0:01:22.170 ******* 2020-03-11T22:28:51.919 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:28:51.920 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:28:51.962 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:51 +0000 (0:00:00.555) 0:01:22.725 ******* 2020-03-11T22:28:52.476 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:28:52.476 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:28:52.951 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:28:53.442 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:28:53.483 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:53 +0000 (0:00:01.521) 0:01:24.246 ******* 2020-03-11T22:28:54.069 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:28:54.071 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.355955", "end": "2020-03-11 22:28:54.039663", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:28:53.683708", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:28:54.073 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:28:54.111 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.627) 0:01:24.874 ******* 2020-03-11T22:28:54.138 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:28:54.139 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.176 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.064) 0:01:24.939 ******* 2020-03-11T22:28:54.223 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:28:54.225 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.236 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.273 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.096) 0:01:25.036 ******* 2020-03-11T22:28:54.299 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:28:54.300 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.337 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.064) 0:01:25.100 ******* 2020-03-11T22:28:54.373 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:54.374 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:28:54.412 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.074) 0:01:25.175 ******* 2020-03-11T22:28:54.448 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:28:54.449 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.487 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.074) 0:01:25.250 ******* 2020-03-11T22:28:54.523 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:28:54.523 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.562 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.074) 0:01:25.324 ******* 2020-03-11T22:28:54.785 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:28:54.786 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006651", "end": "2020-03-11 22:28:54.755908", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:28:54.749257", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:28:54.827 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.265) 0:01:25.590 ******* 2020-03-11T22:28:54.853 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:28:54.854 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:54.889 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:54 +0000 (0:00:00.062) 0:01:25.652 ******* 2020-03-11T22:28:57.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:28:57.762 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:28:57.800 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:57 +0000 (0:00:02.911) 0:01:28.563 ******* 2020-03-11T22:29:00.840 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:29:00.875 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:03.075) 0:01:31.638 ******* 2020-03-11T22:29:06.173 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:29:06.174 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:05.070191", "end": "2020-03-11 22:29:06.143941", "rc": 0, "start": "2020-03-11 22:29:01.073750", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 138 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 129 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 76 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 165 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:29:01 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:29:06.176 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 138 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 129 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 76 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 165 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:29:01 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:29:06.212 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:06 +0000 (0:00:05.336) 0:01:36.975 ******* 2020-03-11T22:29:14.710 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:29:14.746 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:08.534) 0:01:45.509 ******* 2020-03-11T22:29:14.783 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:29:14.783 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.830 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.083) 0:01:45.593 ******* 2020-03-11T22:29:15.848 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:29:15.850 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:29:15.901 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:01.070) 0:01:46.664 ******* 2020-03-11T22:29:15.937 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:15.939 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:15.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:00.074) 0:01:46.738 ******* 2020-03-11T22:29:16.328 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:29:16.329 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service system.slice sysinit.target basic.target polkit.service dbus.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ebtables.service iptables.service shutdown.target ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:29:16.331 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", 2020-03-11T22:29:16.332 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:29:16.334 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:29:16.367 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.392) 0:01:47.130 ******* 2020-03-11T22:29:16.691 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:29:16.693 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:29:16.730 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.362) 0:01:47.493 ******* 2020-03-11T22:29:17.512 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:29:17.514 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:29:17.551 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.820) 0:01:48.314 ******* 2020-03-11T22:29:17.577 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:29:17.579 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.615 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.063) 0:01:48.377 ******* 2020-03-11T22:29:17.640 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:29:17.678 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.062) 0:01:48.440 ******* 2020-03-11T22:29:17.704 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:29:17.706 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.744 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.065) 0:01:48.506 ******* 2020-03-11T22:29:17.769 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:29:17.771 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.809 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.065) 0:01:48.572 ******* 2020-03-11T22:29:17.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:29:17.849 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.859 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.911 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.101) 0:01:48.674 ******* 2020-03-11T22:29:17.937 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:29:17.939 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.985 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.074) 0:01:48.748 ******* 2020-03-11T22:29:18.012 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:29:18.053 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.067) 0:01:48.816 ******* 2020-03-11T22:29:18.082 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:29:18.083 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.120 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.066) 0:01:48.883 ******* 2020-03-11T22:29:18.148 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:29:18.150 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.186 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.066) 0:01:48.949 ******* 2020-03-11T22:29:18.213 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:29:18.215 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.250 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.063) 0:01:49.013 ******* 2020-03-11T22:29:18.276 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:18.278 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.319 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.067) 0:01:49.081 ******* 2020-03-11T22:29:18.345 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:29:18.346 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.385 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.066) 0:01:49.148 ******* 2020-03-11T22:29:18.411 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:18.412 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.450 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.064) 0:01:49.213 ******* 2020-03-11T22:29:18.476 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:29:18.477 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.514 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.063) 0:01:49.276 ******* 2020-03-11T22:29:18.539 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:29:18.541 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.064) 0:01:49.341 ******* 2020-03-11T22:29:18.604 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:29:18.605 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.644 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.065) 0:01:49.407 ******* 2020-03-11T22:29:18.680 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:29:18.681 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.718 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.074) 0:01:49.481 ******* 2020-03-11T22:29:18.743 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:29:18.744 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.800 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.082) 0:01:49.563 ******* 2020-03-11T22:29:18.826 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:29:18.828 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.866 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.065) 0:01:49.629 ******* 2020-03-11T22:29:18.892 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:29:18.894 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.929 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.062) 0:01:49.692 ******* 2020-03-11T22:29:18.965 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:29:18.967 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.015 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.086) 0:01:49.778 ******* 2020-03-11T22:29:19.042 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:29:19.043 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.069) 0:01:49.848 ******* 2020-03-11T22:29:19.111 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:29:19.113 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.151 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.065) 0:01:49.914 ******* 2020-03-11T22:29:19.188 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.230 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.079) 0:01:49.993 ******* 2020-03-11T22:29:19.258 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:29:19.260 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.299 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.068) 0:01:50.062 ******* 2020-03-11T22:29:19.325 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:29:19.327 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.377 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.077) 0:01:50.140 ******* 2020-03-11T22:29:19.402 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:29:19.404 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.062) 0:01:50.202 ******* 2020-03-11T22:29:19.465 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:29:19.467 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.502 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.061) 0:01:50.264 ******* 2020-03-11T22:29:19.537 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:29:19.538 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.077) 0:01:50.341 ******* 2020-03-11T22:29:19.625 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.660 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.081) 0:01:50.423 ******* 2020-03-11T22:29:19.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:19.688 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.724 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.063) 0:01:50.487 ******* 2020-03-11T22:29:19.761 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:19.762 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.812 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.087) 0:01:50.574 ******* 2020-03-11T22:29:19.838 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:19.839 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:19.876 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.064) 0:01:50.638 ******* 2020-03-11T22:29:20.120 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:20.122 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:29:20.181 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.302) 0:01:50.941 ******* 2020-03-11T22:29:20.402 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:20.402 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:20.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.260) 0:01:51.202 ******* 2020-03-11T22:29:20.656 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:20.657 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:29:20.694 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.255) 0:01:51.457 ******* 2020-03-11T22:29:20.720 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:20.721 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.757 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:01:51.520 ******* 2020-03-11T22:29:20.782 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:20.784 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.819 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:01:51.582 ******* 2020-03-11T22:29:20.845 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:20.847 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:01:51.645 ******* 2020-03-11T22:29:20.908 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.945 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:01:51.708 ******* 2020-03-11T22:29:20.971 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:20.972 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.009 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.063) 0:01:51.771 ******* 2020-03-11T22:29:21.035 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:21.036 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.074 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:51.836 ******* 2020-03-11T22:29:21.099 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:21.101 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.138 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:51.901 ******* 2020-03-11T22:29:21.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:21.166 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.205 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:51.965 ******* 2020-03-11T22:29:21.241 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:21.242 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.286 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.083) 0:01:52.049 ******* 2020-03-11T22:29:21.325 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:29:21.326 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.384 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.096) 0:01:52.146 ******* 2020-03-11T22:29:21.413 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:29:21.423 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.462 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.078) 0:01:52.225 ******* 2020-03-11T22:29:21.488 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:21.489 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.525 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.063) 0:01:52.288 ******* 2020-03-11T22:29:21.551 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:21.552 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.596 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.070) 0:01:52.359 ******* 2020-03-11T22:29:21.622 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:29:21.624 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.680 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.084) 0:01:52.443 ******* 2020-03-11T22:29:21.709 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:29:21.710 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.746 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.065) 0:01:52.509 ******* 2020-03-11T22:29:21.773 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:29:21.774 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.809 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.063) 0:01:52.572 ******* 2020-03-11T22:29:21.836 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:29:21.838 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:52.637 ******* 2020-03-11T22:29:21.910 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:29:21.911 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.945 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.071) 0:01:52.708 ******* 2020-03-11T22:29:21.971 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:29:21.972 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.007 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:52.770 ******* 2020-03-11T22:29:22.033 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:29:22.034 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.070 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:52.833 ******* 2020-03-11T22:29:22.096 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:29:22.097 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.061) 0:01:52.895 ******* 2020-03-11T22:29:22.159 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:29:22.160 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.195 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:52.958 ******* 2020-03-11T22:29:22.221 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:22.223 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:29:22.259 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.063) 0:01:53.022 ******* 2020-03-11T22:29:22.306 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:29:22.307 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.327 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:29:22.338 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.349 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.360 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.371 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.383 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.422 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.163) 0:01:53.185 ******* 2020-03-11T22:29:22.646 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:29:22.647 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009592", "end": "2020-03-11 22:29:22.620514", "rc": 0, "start": "2020-03-11 22:29:22.610922", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:29:22.697 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.274) 0:01:53.460 ******* 2020-03-11T22:29:22.744 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:29:22.746 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.793 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.095) 0:01:53.556 ******* 2020-03-11T22:29:22.831 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:29:22.832 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.879 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.083) 0:01:53.640 ******* 2020-03-11T22:29:22.905 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:29:22.906 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.942 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.065) 0:01:53.705 ******* 2020-03-11T22:29:22.969 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:29:22.971 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:23.010 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.067) 0:01:53.772 ******* 2020-03-11T22:29:24.056 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:29:24.058 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:29:24.095 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:01.085) 0:01:54.858 ******* 2020-03-11T22:29:24.145 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:24.146 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={u'block_used': 12964198, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130910, u'size_available': 930328207360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130910, "block_size": 4096, "block_total": 240095108, "block_used": 12964198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328207360, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:29:24.194 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.098) 0:01:54.956 ******* 2020-03-11T22:29:24.417 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:29:24.419 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005236", "end": "2020-03-11 22:29:24.389926", "rc": 0, "start": "2020-03-11 22:29:24.384690", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:24.454 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.259) 0:01:55.216 ******* 2020-03-11T22:29:24.630 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:29:24.631 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'block_used': 12964198, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130910, u'size_available': 930328207360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130910, "block_size": 4096, "block_total": 240095108, "block_used": 12964198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328207360, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.687 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.233) 0:01:55.450 ******* 2020-03-11T22:29:24.718 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:29:24.762 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.074) 0:01:55.525 ******* 2020-03-11T22:29:24.956 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:29:24.958 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HPNL', u'wwn-0x5000c500871a9bab'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HPNL', u'wwn': u'0x5000c500871a9bab', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', u'wwn-0x5000c500871a9bab-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:24.959 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:26.541 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-hTqCuP-fWmR-jPwk-8z6e-WayJ-EcFn-cDxilm', u'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', u'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.375475", "end": "2020-03-11 22:29:26.501441", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-hTqCuP-fWmR-jPwk-8z6e-WayJ-EcFn-cDxilm", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:25.125966", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGP 2020-03-11T22:29:26.543 INFO:teuthology.task.ansible.out:T data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:27.796 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetmQYbhFdZHNCf90KfYBuekF3KtNq31Pzj'], u'uuids': [u'b6b8cc72-504e-45f9-bf75-4f08b24d1e64']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.037736", "end": "2020-03-11 22:29:27.758266", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetmQYbhFdZHNCf90KfYBuekF3KtNq31Pzj"], "labels": [], "masters": [], "uuids": ["b6b8cc72-504e-45f9-bf75-4f08b24d1e64"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:26.720530", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:29.057 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetSbFBblwVV7nMp3z0M21V1IZqGbc0VoL4'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.040413", "end": "2020-03-11 22:29:29.024347", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetSbFBblwVV7nMp3z0M21V1IZqGbc0VoL4"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:27.983934", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:30.344 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetUz824gCk2aY4A9bNIed6HkR4RyNWWCsS'], u'uuids': [u'063164f8-ddbf-4abb-90e8-301546476a72']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.070291", "end": "2020-03-11 22:29:30.309165", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetUz824gCk2aY4A9bNIed6HkR4RyNWWCsS"], "labels": [], "masters": [], "uuids": ["063164f8-ddbf-4abb-90e8-301546476a72"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:29.238874", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:31.588 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetqSYs03VVj7X5qvZNlYF9ZRQkXFSNwGje'], u'uuids': [u'14879d0b-727b-483b-b7d5-6c0b3e5b3f17']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.037583", "end": "2020-03-11 22:29:31.549665", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetqSYs03VVj7X5qvZNlYF9ZRQkXFSNwGje"], "labels": [], "masters": [], "uuids": ["14879d0b-727b-483b-b7d5-6c0b3e5b3f17"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:30.512082", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:32.983 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetMqezCEzRUC5VNKLj7BdPbG1MM2BBbwB0'], u'uuids': [u'496e7f8f-2cd7-45d2-84ae-414ea4555a00']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.061997", "end": "2020-03-11 22:29:32.824786", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetMqezCEzRUC5VNKLj7BdPbG1MM2BBbwB0"], "labels": [], "masters": [], "uuids": ["496e7f8f-2cd7-45d2-84ae-414ea4555a00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:31.762789", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:33.025 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:08.262) 0:02:03.788 ******* 2020-03-11T22:29:33.061 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:29:33.062 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HPNL', u'wwn-0x5000c500871a9bab'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HPNL', u'wwn': u'0x5000c500871a9bab', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', u'wwn-0x5000c500871a9bab-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:33.064 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.312 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-hTqCuP-fWmR-jPwk-8z6e-WayJ-EcFn-cDxilm', u'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', u'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036772", "end": "2020-03-11 22:29:33.288997", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-hTqCuP-fWmR-jPwk-8z6e-WayJ-EcFn-cDxilm", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:33.252225", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2020-03-11T22:29:33.314 INFO:teuthology.task.ansible.out: 0.033985 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.033985 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:33.605 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetmQYbhFdZHNCf90KfYBuekF3KtNq31Pzj'], u'uuids': [u'b6b8cc72-504e-45f9-bf75-4f08b24d1e64']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.095575", "end": "2020-03-11 22:29:33.579806", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetmQYbhFdZHNCf90KfYBuekF3KtNq31Pzj"], "labels": [], "masters": [], "uuids": ["b6b8cc72-504e-45f9-bf75-4f08b24d1e64"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:33.484231", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0928749 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0928749 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:33.936 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetSbFBblwVV7nMp3z0M21V1IZqGbc0VoL4'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.094972", "end": "2020-03-11 22:29:33.902431", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetSbFBblwVV7nMp3z0M21V1IZqGbc0VoL4"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:33.807459", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0923484 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0923484 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:34.220 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetUz824gCk2aY4A9bNIed6HkR4RyNWWCsS'], u'uuids': [u'063164f8-ddbf-4abb-90e8-301546476a72']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.095448", "end": "2020-03-11 22:29:34.195432", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetUz824gCk2aY4A9bNIed6HkR4RyNWWCsS"], "labels": [], "masters": [], "uuids": ["063164f8-ddbf-4abb-90e8-301546476a72"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:34.099984", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0928124 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0928124 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:34.511 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetqSYs03VVj7X5qvZNlYF9ZRQkXFSNwGje'], u'uuids': [u'14879d0b-727b-483b-b7d5-6c0b3e5b3f17']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.094028", "end": "2020-03-11 22:29:34.484126", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetqSYs03VVj7X5qvZNlYF9ZRQkXFSNwGje"], "labels": [], "masters": [], "uuids": ["14879d0b-727b-483b-b7d5-6c0b3e5b3f17"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:34.390098", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0913982 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0913982 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:34.811 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetMqezCEzRUC5VNKLj7BdPbG1MM2BBbwB0'], u'uuids': [u'496e7f8f-2cd7-45d2-84ae-414ea4555a00']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.094362", "end": "2020-03-11 22:29:34.783840", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RWdd4MLHmUadEvqiT3y8ZBWlWYWA1YetMqezCEzRUC5VNKLj7BdPbG1MM2BBbwB0"], "labels": [], "masters": [], "uuids": ["496e7f8f-2cd7-45d2-84ae-414ea 2020-03-11T22:29:34.813 INFO:teuthology.task.ansible.out:4555a00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:34.689478", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0916832 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0916832 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:34.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:01.849) 0:02:05.637 ******* 2020-03-11T22:29:35.173 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:29:35.175 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.084672", "end": "2020-03-11 22:29:35.138439", "rc": 0, "start": "2020-03-11 22:29:35.053767", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:35.210 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.336) 0:02:05.973 ******* 2020-03-11T22:29:35.464 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:29:35.466 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045631", "end": "2020-03-11 22:29:35.435107", "rc": 0, "start": "2020-03-11 22:29:35.389476", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:35.503 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.292) 0:02:06.266 ******* 2020-03-11T22:29:35.529 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:29:35.573 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.070) 0:02:06.336 ******* 2020-03-11T22:29:35.602 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:29:35.602 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.658 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.084) 0:02:06.420 ******* 2020-03-11T22:29:35.684 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:29:35.685 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.727 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.069) 0:02:06.490 ******* 2020-03-11T22:29:35.754 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:35.755 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.795 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.067) 0:02:06.558 ******* 2020-03-11T22:29:35.821 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:29:35.823 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.078) 0:02:06.637 ******* 2020-03-11T22:29:36.097 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:29:36.098 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:36.150 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.276) 0:02:06.913 ******* 2020-03-11T22:29:36.209 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:36.210 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={u'block_used': 12964198, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130910, u'size_available': 930328207360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130910, "block_size": 4096, "block_total": 240095108, "block_used": 12964198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328207360, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.100) 0:02:07.014 ******* 2020-03-11T22:29:36.276 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:29:36.278 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.316 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.065) 0:02:07.079 ******* 2020-03-11T22:29:36.342 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:36.344 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.387 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.070) 0:02:07.150 ******* 2020-03-11T22:29:36.414 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:36.415 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.463 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.076) 0:02:07.226 ******* 2020-03-11T22:29:36.490 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:29:36.492 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.533 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.069) 0:02:07.296 ******* 2020-03-11T22:29:36.569 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:29:36.571 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.608 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.074) 0:02:07.370 ******* 2020-03-11T22:29:36.644 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:36.646 INFO:teuthology.task.ansible.out:TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.688 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.080) 0:02:07.451 ******* 2020-03-11T22:29:37.481 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:29:37.482 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:29:37.527 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.838) 0:02:08.290 ******* 2020-03-11T22:29:38.056 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:29:38.058 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:38.447 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:38.832 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:39.177 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:39.542 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:39.583 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:02.056) 0:02:10.346 ******* 2020-03-11T22:29:39.821 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:39.823 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:29:39.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.280) 0:02:10.627 ******* 2020-03-11T22:29:39.900 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:29:39.901 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.109 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:40.306 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:40.507 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:40.706 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:40.780 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.916) 0:02:11.543 ******* 2020-03-11T22:29:41.022 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:29:41.023 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:41.065 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.284) 0:02:11.828 ******* 2020-03-11T22:29:41.101 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:29:41.103 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:29:41.146 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.081) 0:02:11.909 ******* 2020-03-11T22:29:41.535 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:29:41.537 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:29:41.577 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.431) 0:02:12.340 ******* 2020-03-11T22:29:42.055 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:29:42.056 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:29:42.097 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.519) 0:02:12.860 ******* 2020-03-11T22:29:42.124 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:29:42.125 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.165 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.067) 0:02:12.928 ******* 2020-03-11T22:29:42.191 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:29:42.193 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.243 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.077) 0:02:13.006 ******* 2020-03-11T22:29:42.272 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:29:42.273 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.309 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.066) 0:02:13.072 ******* 2020-03-11T22:29:42.336 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:29:42.338 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.375 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.065) 0:02:13.137 ******* 2020-03-11T22:29:42.401 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:29:42.402 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.063) 0:02:13.201 ******* 2020-03-11T22:29:42.475 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:29:42.476 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.515 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.076) 0:02:13.278 ******* 2020-03-11T22:29:42.541 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:29:42.543 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.588 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.073) 0:02:13.351 ******* 2020-03-11T22:29:42.614 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:29:42.616 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.654 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.065) 0:02:13.417 ******* 2020-03-11T22:29:42.680 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:29:42.682 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.724 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.069) 0:02:13.486 ******* 2020-03-11T22:29:42.750 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:29:42.752 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.791 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.067) 0:02:13.554 ******* 2020-03-11T22:29:42.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:29:42.819 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.855 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.064) 0:02:13.618 ******* 2020-03-11T22:29:43.396 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:29:43.398 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:29:43.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:43 +0000 (0:00:00.583) 0:02:14.202 ******* 2020-03-11T22:29:43.775 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:29:43.777 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:26:47 UTC", "ActiveEnterTimestampMonotonic": "14336696", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service ntpd.service sysinit.target systemd-journald.socket -.mount tmp.mount sntp.service basic.target systemd-tmpfiles-setup.service system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:26:45 UTC", "AssertTimestampMonotonic": "12914718", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:26:45 UTC", "ConditionTimestampMonotonic": "12914695", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:29:43.779 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2318", "ExecMainStartTimestamp": "Wed 2020-03-11 22:26:45 UTC", "ExecMainStartTimestampMonotonic": "13071233", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:26:45 UTC", "InactiveExitTimestampMonotonic": "12991888", "InvocationID": "ae2f8c9213db4181b67c753517b7d2ad", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:29:43.780 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", "LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2318", "MemoryAccounting": "yes", "MemoryCurrent": "2883584", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:29:43.782 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:26:47 UTC", "StateChangeTimestampMonotonic": "14336696", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:26:45 UTC", "WatchdogTimestampMonotonic": "13071241", "WatchdogUSec": "0"}} 2020-03-11T22:29:43.827 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:43 +0000 (0:00:00.388) 0:02:14.590 ******* 2020-03-11T22:29:43.853 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:29:43.855 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:43.899 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:43 +0000 (0:00:00.071) 0:02:14.662 ******* 2020-03-11T22:29:43.936 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:29:43.938 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:43.983 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:43 +0000 (0:00:00.084) 0:02:14.746 ******* 2020-03-11T22:29:44.009 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:44.011 INFO:teuthology.task.ansible.out:TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:44.053 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.069) 0:02:14.815 ******* 2020-03-11T22:29:44.274 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:29:44.275 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:29:44.334 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.281) 0:02:15.097 ******* 2020-03-11T22:29:44.843 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:29:44.845 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:29:44.881 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.547) 0:02:15.644 ******* 2020-03-11T22:29:46.079 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:29:46.080 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:29:46.135 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:01.254) 0:02:16.898 ******* 2020-03-11T22:29:46.162 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:29:46.163 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.218 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.082) 0:02:16.981 ******* 2020-03-11T22:29:46.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:29:46.256 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-03-11T22:29:46.291 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.073) 0:02:17.054 ******* 2020-03-11T22:29:46.495 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:29:46.497 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:29:46.541 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.249) 0:02:17.304 ******* 2020-03-11T22:29:46.568 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:29:46.569 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.607 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.066) 0:02:17.370 ******* 2020-03-11T22:29:46.643 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:29:46.645 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:29:46.686 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.078) 0:02:17.449 ******* 2020-03-11T22:29:46.712 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:29:46.713 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.751 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.065) 0:02:17.514 ******* 2020-03-11T22:29:46.968 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:29:46.970 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:29:47.006 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.254) 0:02:17.768 ******* 2020-03-11T22:29:47.214 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:29:47.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.245) 0:02:18.014 ******* 2020-03-11T22:29:47.277 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:29:47.279 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.328 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.076) 0:02:18.091 ******* 2020-03-11T22:29:47.367 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:29:47.368 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.369 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.370 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.381 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.382 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.423 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.094) 0:02:18.186 ******* 2020-03-11T22:29:47.459 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:29:47.461 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.471 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.472 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.483 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.524 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.101) 0:02:18.287 ******* 2020-03-11T22:29:47.552 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.607 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.083) 0:02:18.370 ******* 2020-03-11T22:29:47.633 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:29:47.635 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.677 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.069) 0:02:18.440 ******* 2020-03-11T22:29:47.704 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:29:47.706 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.763 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.086) 0:02:18.526 ******* 2020-03-11T22:29:47.800 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:47.801 INFO:teuthology.task.ansible.out:TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.849 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.085) 0:02:18.611 ******* 2020-03-11T22:29:47.874 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:29:47.876 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:47.917 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.068) 0:02:18.680 ******* 2020-03-11T22:29:47.953 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:29:47.955 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:48.010 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.082) 0:02:18.763 ******* 2020-03-11T22:29:48.037 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:29:48.039 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:48.083 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:48 +0000 (0:00:00.083) 0:02:18.846 ******* 2020-03-11T22:29:48.119 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:29:48.121 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:48.172 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:48 +0000 (0:00:00.085) 0:02:18.932 ******* 2020-03-11T22:29:48.410 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:29:48.410 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005418", "end": "2020-03-11 22:29:48.369177", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:29:48.363759", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:48.610 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003077", "end": "2020-03-11 22:29:48.574372", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:29:48.571295", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:48.611 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:48.807 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002961", "end": "2020-03-11 22:29:48.770478", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:29:48.767517", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:49.035 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026764", "end": "2020-03-11 22:29:48.997704", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:29:48.970940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:49.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.916) 0:02:19.848 ******* 2020-03-11T22:29:49.113 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:29:49.153 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.067) 0:02:19.916 ******* 2020-03-11T22:29:49.364 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:29:49.365 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:29:49.488 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-11T22:29:49.490 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-11T22:29:49.492 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-03-11T22:29:49.494 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi045.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.339) 0:02:20.256 ******* =============================================================================== common : nrpe - Remove SELinux policy package -------------------------- 17.28s common : nrpe - Load SELinux policy package ---------------------------- 16.99s testnode : Install packages --------------------------------------------- 8.53s testnode : Zap all non-root disks --------------------------------------- 8.26s testnode : Base group workaround for BZ1782899 -------------------------- 5.34s common : Install Kerberos Packages (RedHat) ----------------------------- 3.28s testnode : Ensure ceph dependency packages are not present. ------------- 3.08s Gathering Facts --------------------------------------------------------- 2.96s testnode : Ensure ceph packages are not present. ------------------------ 2.91s 2020-03-11T22:29:49.494 INFO:teuthology.task.ansible.out:testnode : Create logical volume(s) ------------------------------------- 2.06s testnode : Rebuild rpmdb ------------------------------------------------ 2.05s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.85s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.58s testnode : Configure local mirrorlists ---------------------------------- 1.54s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.52s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.49s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.42s common : Set the hardware clock ----------------------------------------- 1.26s 2020-03-11T22:29:49.494 INFO:teuthology.task.ansible.out:testnode : Install packages via pip ------------------------------------- 1.25s testnode : Set the system's hostname -------------------- 2020-03-11T22:29:49.496 INFO:teuthology.task.ansible.out:---------------- 1.18s 2020-03-11T22:29:49.771 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2020-03-11T22:29:49.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:49.774 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi045.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:49.848 INFO:teuthology.orchestra.run.smithi045:> true 2020-03-11T22:29:49.939 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:29:49.957 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:29:49.957 INFO:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:29:50.018 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:29:50.040 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:29:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2020-03-11T22:29:50.103 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2020-03-11T22:29:50.124 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:29:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:29:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:bash: ntpq: command not found 2020-03-11T22:29:50.213 INFO:teuthology.orchestra.run.smithi045.stdout:210 Number of sources = 4 2020-03-11T22:29:50.213 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:29:50.213 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2020-03-11T22:29:50.214 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:50.214 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:50.214 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:50.214 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:50.214 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:29:50.225 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:29:50.225 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:29:50.225 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:29:50.226 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:29:50.238 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:29:50.238 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:29:50.238 INFO:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2020-03-11T22:29:50.295 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS 2020-03-11T22:29:50.296 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:29:50.296 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:29:50.296 INFO:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2020-03-11T22:29:50.375 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS 2020-03-11T22:29:50.376 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:29:50.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:29:50.604 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:29:50.605 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:29:50.791 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:29:50.791 INFO:teuthology.orchestra.run.smithi045:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:29:50.837 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:29:50.837 INFO:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:29:50.910 INFO:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:29:50.998 INFO:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:29:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2020-03-11T22:29:51.030 INFO:teuthology.orchestra.run.smithi045:> sudo yum clean all 2020-03-11T22:29:51.400 INFO:teuthology.orchestra.run.smithi045.stdout:239 files removed 2020-03-11T22:29:51.430 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2020-03-11T22:29:52.174 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 183 kB/s | 69 kB 00:00 2020-03-11T22:29:52.512 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 44 kB/s | 14 kB 00:00 2020-03-11T22:29:52.808 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 5.2 kB/s | 1.4 kB 00:00 2020-03-11T22:29:52.966 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-8 - AppStream 47 MB/s | 6.5 MB 00:00 2020-03-11T22:29:54.200 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-8 - Base 37 MB/s | 5.0 MB 00:00 2020-03-11T22:29:55.028 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:29:55.182 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for ceph-el8 owned by ktdreyer 359 kB/s | 38 kB 00:00 2020-03-11T22:29:55.636 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 15 MB/s | 6.0 MB 00:00 2020-03-11T22:29:56.927 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 5.5 MB/s | 30 kB 00:00 2020-03-11T22:29:57.951 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:29:57.954 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:29:57.954 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:29:57.955 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:29:57.956 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:29:57.957 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:29:57.958 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:29:57.959 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:29:57.959 INFO:teuthology.orchestra.run.smithi045.stdout:Install 21 Packages 2020-03-11T22:29:57.959 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2020-03-11T22:29:57.959 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:29:57.959 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 51 M 2020-03-11T22:29:57.960 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:29:58.895 INFO:teuthology.orchestra.run.smithi045.stdout:(1/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 10 MB/s | 9.8 MB 00:00 2020-03-11T22:29:58.944 INFO:teuthology.orchestra.run.smithi045.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 502 kB/s | 25 kB 00:00 2020-03-11T22:29:58.998 INFO:teuthology.orchestra.run.smithi045.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 9.3 MB/s | 507 kB 00:00 2020-03-11T22:29:59.046 INFO:teuthology.orchestra.run.smithi045.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 7.6 MB/s | 377 kB 00:00 2020-03-11T22:29:59.154 INFO:teuthology.orchestra.run.smithi045.stdout:(5/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 4.6 MB/s | 5.5 MB 00:01 2020-03-11T22:29:59.221 INFO:teuthology.orchestra.run.smithi045.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 590 kB/s | 40 kB 00:00 2020-03-11T22:29:59.288 INFO:teuthology.orchestra.run.smithi045.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 741 kB/s | 49 kB 00:00 2020-03-11T22:29:59.358 INFO:teuthology.orchestra.run.smithi045.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 2.2 MB/s | 162 kB 00:00 2020-03-11T22:29:59.450 INFO:teuthology.orchestra.run.smithi045.stdout:(9/23): librgw2-15.1.0-1941.g4acc112.el8.x86_64 15 MB/s | 6.2 MB 00:00 2020-03-11T22:29:59.452 INFO:teuthology.orchestra.run.smithi045.stdout:(10/23): python3-rados-15.1.0-1941.g4acc112.el8 3.7 MB/s | 352 kB 00:00 2020-03-11T22:29:59.504 INFO:teuthology.orchestra.run.smithi045.stdout:(11/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 5.5 MB/s | 309 kB 00:00 2020-03-11T22:29:59.513 INFO:teuthology.orchestra.run.smithi045.stdout:(12/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 1.8 MB/s | 113 kB 00:00 2020-03-11T22:29:59.540 INFO:teuthology.orchestra.run.smithi045.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 9.7 MB/s | 353 kB 00:00 2020-03-11T22:29:59.542 INFO:teuthology.orchestra.run.smithi045.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 28 MB/s | 44 kB 00:00 2020-03-11T22:29:59.544 INFO:teuthology.orchestra.run.smithi045.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 24 MB/s | 47 kB 00:00 2020-03-11T22:29:59.546 INFO:teuthology.orchestra.run.smithi045.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.6 MB/s | 259 kB 00:00 2020-03-11T22:29:59.621 INFO:teuthology.orchestra.run.smithi045.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 789 kB/s | 59 kB 00:00 2020-03-11T22:29:59.659 INFO:teuthology.orchestra.run.smithi045.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2020-03-11T22:29:59.664 INFO:teuthology.orchestra.run.smithi045.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 33 MB/s | 179 kB 00:00 2020-03-11T22:29:59.675 INFO:teuthology.orchestra.run.smithi045.stdout:(20/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.4 MB/s | 315 kB 00:00 2020-03-11T22:29:59.822 INFO:teuthology.orchestra.run.smithi045.stdout:(21/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 13 MB/s | 1.9 MB 00:00 2020-03-11T22:30:00.063 INFO:teuthology.orchestra.run.smithi045.stdout:(22/23): ceph-common-15.1.0-1941.g4acc112.el8.x 9.8 MB/s | 21 MB 00:02 2020-03-11T22:30:00.145 INFO:teuthology.orchestra.run.smithi045.stdout:(23/23): librados2-15.1.0-1941.g4acc112.el8.x86 7.4 MB/s | 3.6 MB 00:00 2020-03-11T22:30:00.148 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:00.149 INFO:teuthology.orchestra.run.smithi045.stdout:Total 23 MB/s | 51 MB 00:02 2020-03-11T22:30:00.155 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:30:00.202 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:30:00.202 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:30:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:30:00.569 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:30:01.699 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:30:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:30:02.044 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:02.478 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:02.819 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:02.900 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:03.106 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:03.227 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:03.489 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:03.677 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:03.742 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:03.921 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:30:04.086 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:04.744 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:04.927 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:04.976 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:05.112 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:30:05.262 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:05.654 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:05.802 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:05.876 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:06.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:30:06.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:30:06.394 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:30:06.572 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:30:06.703 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:30:10.452 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:10.668 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:11.391 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:11.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:12.009 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:28.902 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /run 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2020-03-11T22:30:28.903 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2020-03-11T22:30:28.904 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2020-03-11T22:30:28.904 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2020-03-11T22:30:28.904 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:30.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:30.193 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:30.194 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:30.384 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:30.384 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:31.072 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:31.073 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:31.074 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:30:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:30:31.076 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:31.076 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:30:31.076 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:30:31.871 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:31.871 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.872 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:30:31.874 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:30:31.875 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:30:31.875 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:30:31.875 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:30:31.875 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:30:31.875 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:31.876 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:30:32.206 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2020-03-11T22:30:32.650 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:36 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:30:33.041 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:30:33.042 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:30:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2020-03-11T22:30:33.044 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:33.045 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 43 M 2020-03-11T22:30:33.045 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 184 M 2020-03-11T22:30:33.045 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:30:33.089 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 3.5 MB/s | 161 kB 00:00 2020-03-11T22:30:33.094 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2020-03-11T22:30:33.096 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 43 MB/s | 298 kB 00:00 2020-03-11T22:30:33.100 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 40 MB/s | 249 kB 00:00 2020-03-11T22:30:33.159 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-11T22:30:35.158 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 20 MB/s | 42 MB 00:02 2020-03-11T22:30:35.161 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:35.161 INFO:teuthology.orchestra.run.smithi045.stdout:Total 20 MB/s | 43 MB 00:02 2020-03-11T22:30:35.167 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:30:35.192 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:30:35.192 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:30:35.464 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:30:35.465 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:30:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:30:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:30:36.315 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:30:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:30:36.647 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:36.712 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:40.979 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:30:41.169 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:41.386 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:41.386 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:30:41.387 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:30:41.387 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:30:41.387 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:30:41.387 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:30:41.718 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:30:41.719 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:41.720 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:30:41.982 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2020-03-11T22:30:42.407 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:46 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:30:42.826 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:30:42.830 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:42.830 INFO:teuthology.orchestra.run.smithi045.stdout: Package ArchVersion Repository Size 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:30:42.831 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:30:42.832 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:30:42.833 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:30:42.833 INFO:teuthology.orchestra.run.smithi045.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:30:42.833 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:30:42.833 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:30:42.833 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:30:42.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:30:42.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:30:42.836 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:42.837 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:30:42.837 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:42.837 INFO:teuthology.orchestra.run.smithi045.stdout:Install 36 Packages 2020-03-11T22:30:42.837 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 29 M 2020-03-11T22:30:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 105 M 2020-03-11T22:30:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:30:42.990 INFO:teuthology.orchestra.run.smithi045.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 42 kB/s | 6.4 kB 00:00 2020-03-11T22:30:43.255 INFO:teuthology.orchestra.run.smithi045.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.4 MB/s | 1.4 MB 00:00 2020-03-11T22:30:43.301 INFO:teuthology.orchestra.run.smithi045.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.4 MB/s | 2.0 MB 00:00 2020-03-11T22:30:43.357 INFO:teuthology.orchestra.run.smithi045.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.9 MB/s | 165 kB 00:00 2020-03-11T22:30:43.393 INFO:teuthology.orchestra.run.smithi045.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.3 MB/s | 214 kB 00:00 2020-03-11T22:30:43.409 INFO:teuthology.orchestra.run.smithi045.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 78 MB/s | 1.5 MB 00:00 2020-03-11T22:30:43.412 INFO:teuthology.orchestra.run.smithi045.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 55 MB/s | 157 kB 00:00 2020-03-11T22:30:43.414 INFO:teuthology.orchestra.run.smithi045.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2020-03-11T22:30:43.415 INFO:teuthology.orchestra.run.smithi045.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 40 MB/s | 54 kB 00:00 2020-03-11T22:30:43.417 INFO:teuthology.orchestra.run.smithi045.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 25 MB/s | 27 kB 00:00 2020-03-11T22:30:43.423 INFO:teuthology.orchestra.run.smithi045.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 73 MB/s | 457 kB 00:00 2020-03-11T22:30:43.426 INFO:teuthology.orchestra.run.smithi045.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 64 MB/s | 195 kB 00:00 2020-03-11T22:30:43.428 INFO:teuthology.orchestra.run.smithi045.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 26 MB/s | 34 kB 00:00 2020-03-11T22:30:43.431 INFO:teuthology.orchestra.run.smithi045.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 51 MB/s | 123 kB 00:00 2020-03-11T22:30:43.433 INFO:teuthology.orchestra.run.smithi045.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 59 MB/s | 176 kB 00:00 2020-03-11T22:30:43.493 INFO:teuthology.orchestra.run.smithi045.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.4 MB/s | 145 kB 00:00 2020-03-11T22:30:43.515 INFO:teuthology.orchestra.run.smithi045.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 17 MB/s | 385 kB 00:00 2020-03-11T22:30:43.523 INFO:teuthology.orchestra.run.smithi045.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.4 MB/s | 11 kB 00:00 2020-03-11T22:30:43.531 INFO:teuthology.orchestra.run.smithi045.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.2 MB/s | 18 kB 00:00 2020-03-11T22:30:43.549 INFO:teuthology.orchestra.run.smithi045.stdout:(20/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 7.8 MB/s | 4.4 MB 00:00 2020-03-11T22:30:43.550 INFO:teuthology.orchestra.run.smithi045.stdout:(21/36): python3-logutils-0.3.5-11.el8.noarch.r 2.6 MB/s | 49 kB 00:00 2020-03-11T22:30:43.568 INFO:teuthology.orchestra.run.smithi045.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 19 MB/s | 283 kB 00:00 2020-03-11T22:30:43.573 INFO:teuthology.orchestra.run.smithi045.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.8 MB/s | 16 kB 00:00 2020-03-11T22:30:43.582 INFO:teuthology.orchestra.run.smithi045.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.2 MB/s | 19 kB 00:00 2020-03-11T22:30:43.592 INFO:teuthology.orchestra.run.smithi045.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.3 MB/s | 24 kB 00:00 2020-03-11T22:30:43.598 INFO:teuthology.orchestra.run.smithi045.stdout:(26/36): python3-more-itertools-7.2.0-3.el8.noa 1.2 MB/s | 59 kB 00:00 2020-03-11T22:30:43.601 INFO:teuthology.orchestra.run.smithi045.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.6 MB/s | 30 kB 00:00 2020-03-11T22:30:43.607 INFO:teuthology.orchestra.run.smithi045.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.9 MB/s | 27 kB 00:00 2020-03-11T22:30:43.609 INFO:teuthology.orchestra.run.smithi045.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.7 MB/s | 23 kB 00:00 2020-03-11T22:30:43.671 INFO:teuthology.orchestra.run.smithi045.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 691 kB/s | 44 kB 00:00 2020-03-11T22:30:43.702 INFO:teuthology.orchestra.run.smithi045.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 1.3 MB/s | 40 kB 00:00 2020-03-11T22:30:43.704 INFO:teuthology.orchestra.run.smithi045.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-11T22:30:43.742 INFO:teuthology.orchestra.run.smithi045.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.4 MB/s | 251 kB 00:00 2020-03-11T22:30:43.748 INFO:teuthology.orchestra.run.smithi045.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 5.1 MB/s | 241 kB 00:00 2020-03-11T22:30:43.761 INFO:teuthology.orchestra.run.smithi045.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-03-11T22:30:44.118 INFO:teuthology.orchestra.run.smithi045.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 19 MB/s | 17 MB 00:00 2020-03-11T22:30:44.122 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:44.123 INFO:teuthology.orchestra.run.smithi045.stdout:Total 23 MB/s | 29 MB 00:01 2020-03-11T22:30:44.145 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:30:44.187 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:30:44.187 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:30:44.502 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:30:44.502 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:30:45.929 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:30:46.293 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:30:46.515 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:30:46.740 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:30:46.914 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:30:47.054 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:30:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:30:47.411 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:30:47.564 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:30:47.709 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:30:47.899 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:30:48.257 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:30:48.471 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:30:48.697 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:30:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:30:49.189 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:30:49.399 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:30:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:30:50.044 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:30:50.413 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:30:50.688 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:30:50.894 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:30:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:30:51.243 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:30:51.489 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:30:51.697 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:30:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:30:52.303 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:30:52.555 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:30:52.894 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:30:53.069 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:30:54.591 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:30:54.762 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:30:55.173 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:30:55.328 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:30:55.531 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:30:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:30:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:30:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:30:56.115 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:30:56.116 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:30:56.118 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:30:56.118 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:30:56.118 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:30:56.118 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:30:56.118 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:30:56.119 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:30:56.120 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:30:56.120 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:30:56.837 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:30:56.933 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:56.933 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:30:56.933 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:56.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:30:56.935 INFO:teuthology.orchestra.run.smithi045.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:30:56.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:30:57.223 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2020-03-11T22:30:57.653 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:30:57.949 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:58.063 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:30:58.064 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:30:58.064 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:30:58.212 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2020-03-11T22:30:58.637 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:30:59.046 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:30:59.046 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:30:59.047 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:30:59.048 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:30:59.048 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:30:59.048 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 36 k 2020-03-11T22:30:59.048 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 129 k 2020-03-11T22:30:59.048 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:30:59.235 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 189 kB/s | 36 kB 00:00 2020-03-11T22:30:59.236 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:59.236 INFO:teuthology.orchestra.run.smithi045.stdout:Total 188 kB/s | 36 kB 00:00 2020-03-11T22:30:59.238 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:30:59.246 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:30:59.246 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:30:59.248 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:30:59.248 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:30:59.290 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:00.703 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:00.967 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:01.182 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:01.183 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:01.314 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:31:01.745 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:02.137 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:02.137 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:02.137 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:02.138 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:02.139 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:02.139 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 203 k 2020-03-11T22:31:02.139 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 743 k 2020-03-11T22:31:02.139 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 688 kB/s | 203 kB 00:00 2020-03-11T22:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:Total 686 kB/s | 203 kB 00:00 2020-03-11T22:31:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:02.653 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:02.819 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:03.147 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:03.400 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:03.721 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2020-03-11T22:31:04.150 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:08 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:04.425 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:04.536 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:04.537 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:04.537 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:04.670 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:31:05.092 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:09 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:31:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:31:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:31:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:31:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:31:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:31:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:Install 5 Packages 2020-03-11T22:31:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 4.2 M 2020-03-11T22:31:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 19 M 2020-03-11T22:31:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2020-03-11T22:31:05.566 INFO:teuthology.orchestra.run.smithi045.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 635 kB/s | 34 kB 00:00 2020-03-11T22:31:05.594 INFO:teuthology.orchestra.run.smithi045.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 6.9 MB/s | 197 kB 00:00 2020-03-11T22:31:05.669 INFO:teuthology.orchestra.run.smithi045.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 120 kB/s | 22 kB 00:00 2020-03-11T22:31:05.906 INFO:teuthology.orchestra.run.smithi045.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 9.1 MB/s | 3.9 MB 00:00 2020-03-11T22:31:05.909 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:05.909 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.7 MB/s | 4.2 MB 00:00 2020-03-11T22:31:05.914 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:05.923 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:05.923 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:06.071 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:06.071 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:06.342 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:06.560 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:31:06.787 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:31:06.976 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:08.249 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:31:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:08.749 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:31:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:31:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:31:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:31:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:08.999 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:09.000 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:09.159 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:31:09.581 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:13 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:09.968 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:09.970 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:09.970 INFO:teuthology.orchestra.run.smithi045.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:09.970 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:09.971 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:09.971 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:31:09.971 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:31:09.971 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:31:09.971 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:31:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:31:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:31:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:31:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:31:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:31:09.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:09.973 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:09.973 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:09.973 INFO:teuthology.orchestra.run.smithi045.stdout:Install 8 Packages 2020-03-11T22:31:09.974 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2020-03-11T22:31:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 205 M 2020-03-11T22:31:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:10.077 INFO:teuthology.orchestra.run.smithi045.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 41 MB/s | 4.3 MB 00:00 2020-03-11T22:31:10.133 INFO:teuthology.orchestra.run.smithi045.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2020-03-11T22:31:10.150 INFO:teuthology.orchestra.run.smithi045.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2020-03-11T22:31:10.151 INFO:teuthology.orchestra.run.smithi045.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 12 MB/s | 225 kB 00:00 2020-03-11T22:31:10.172 INFO:teuthology.orchestra.run.smithi045.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 30 MB/s | 638 kB 00:00 2020-03-11T22:31:10.181 INFO:teuthology.orchestra.run.smithi045.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 18 MB/s | 169 kB 00:00 2020-03-11T22:31:10.366 INFO:teuthology.orchestra.run.smithi045.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 67 MB/s | 15 MB 00:00 2020-03-11T22:31:10.655 INFO:teuthology.orchestra.run.smithi045.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 13 MB/s | 8.6 MB 00:00 2020-03-11T22:31:10.658 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:10.658 INFO:teuthology.orchestra.run.smithi045.stdout:Total 53 MB/s | 37 MB 00:00 2020-03-11T22:31:10.669 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:10.690 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:10.690 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:11.001 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:11.001 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:11.563 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:11.717 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:11.863 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:12.591 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:14.208 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:14.958 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:15.275 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:15.475 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:17.561 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:18.918 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:31:19.099 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:31:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:31:19.937 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:31:19.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:19.939 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:20.090 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:31:20.518 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:20.922 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:20.924 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:20.924 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2020-03-11T22:31:20.924 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:20.924 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:20.924 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-noarch 24 k 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:31:20.925 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:31:20.926 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:20.927 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:20.927 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:20.927 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2020-03-11T22:31:20.927 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:20.928 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.9 M 2020-03-11T22:31:20.928 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 23 M 2020-03-11T22:31:20.928 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:20.958 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 879 kB/s | 27 kB 00:00 2020-03-11T22:31:20.961 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 35 MB/s | 126 kB 00:00 2020-03-11T22:31:20.962 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 581 kB/s | 20 kB 00:00 2020-03-11T22:31:20.969 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 14 MB/s | 110 kB 00:00 2020-03-11T22:31:20.970 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 18 MB/s | 155 kB 00:00 2020-03-11T22:31:20.971 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 32 MB/s | 43 kB 00:00 2020-03-11T22:31:21.017 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 351 kB/s | 16 kB 00:00 2020-03-11T22:31:21.018 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 695 kB/s | 33 kB 00:00 2020-03-11T22:31:21.049 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-03-11T22:31:21.064 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.5 MB/s | 73 kB 00:00 2020-03-11T22:31:21.081 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.6 MB/s | 61 kB 00:00 2020-03-11T22:31:21.115 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 130 kB/s | 24 kB 00:00 2020-03-11T22:31:21.128 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 10 MB/s | 1.1 MB 00:00 2020-03-11T22:31:21.129 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:21.130 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.1 MB/s | 1.9 MB 00:00 2020-03-11T22:31:21.137 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:21.152 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:21.153 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:21.230 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:21.230 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:21.988 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:31:22.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:31:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:31:22.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:31:22.807 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:31:23.024 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:31:23.304 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:31:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:31:24.685 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:31:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:31:25.676 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:31:26.209 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:31:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:26.603 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:26.603 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:31:26.603 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:31:26.603 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:31:26.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:31:26.605 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:31:26.930 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:31:26.930 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:26.930 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:26.930 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:26.930 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:31:26.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:31:26.932 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:31:26.932 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:31:26.932 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:31:26.932 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:26.932 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:27.109 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:31:27.539 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:27.932 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:27.933 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:27.933 INFO:teuthology.orchestra.run.smithi045.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:27.933 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:27.933 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:27.933 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:27.934 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:27.935 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2020-03-11T22:31:27.935 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:27.935 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 461 k 2020-03-11T22:31:27.935 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 1.4 M 2020-03-11T22:31:27.936 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:28.004 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 696 kB/s | 48 kB 00:00 2020-03-11T22:31:28.005 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 4.9 MB/s | 356 kB 00:00 2020-03-11T22:31:28.089 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 251 kB/s | 21 kB 00:00 2020-03-11T22:31:28.323 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 94 kB/s | 36 kB 00:00 2020-03-11T22:31:28.324 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:28.324 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.2 MB/s | 461 kB 00:00 2020-03-11T22:31:28.328 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:28.336 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:28.336 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:28.366 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:28.366 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:28.541 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:28.805 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:31:29.082 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:29.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:29.398 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:31:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:31:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:29.914 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:30.073 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2020-03-11T22:31:30.502 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:30.899 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:30.899 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2020-03-11T22:31:30.900 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:31:30.901 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:31:30.901 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:30.901 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:30.901 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:30.901 INFO:teuthology.orchestra.run.smithi045.stdout:Install 3 Packages 2020-03-11T22:31:30.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:30.902 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 676 k 2020-03-11T22:31:30.902 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.1 M 2020-03-11T22:31:30.903 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:30.932 INFO:teuthology.orchestra.run.smithi045.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-11T22:31:30.936 INFO:teuthology.orchestra.run.smithi045.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 612 kB/s | 21 kB 00:00 2020-03-11T22:31:31.241 INFO:teuthology.orchestra.run.smithi045.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.6 MB/s | 572 kB 00:00 2020-03-11T22:31:31.243 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:31.243 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.9 MB/s | 676 kB 00:00 2020-03-11T22:31:31.247 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:31.270 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:31.270 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:31.340 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:31.341 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:31.626 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:31.626 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:31:31.626 INFO:teuthology.orchestra.run.smithi045.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:31:31.627 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:32.016 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:32.205 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:32.495 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:32.496 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:31:32.496 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:31:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:32.813 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:33.100 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2020-03-11T22:31:33.536 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:33.952 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:33.953 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:33.953 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2020-03-11T22:31:33.953 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:33.954 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:33.954 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:33.954 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:33.954 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:33.955 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:33.955 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:33.955 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:33.955 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 141 k 2020-03-11T22:31:33.956 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 487 k 2020-03-11T22:31:33.956 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:34.179 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 623 kB/s | 141 kB 00:00 2020-03-11T22:31:34.180 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:34.180 INFO:teuthology.orchestra.run.smithi045.stdout:Total 620 kB/s | 141 kB 00:00 2020-03-11T22:31:34.182 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:34.190 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:34.190 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:34.244 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:34.244 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:34.354 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:34.512 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:34.750 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:35.020 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:35.021 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:35.334 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2020-03-11T22:31:35.767 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:36.070 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:36.186 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:36.187 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:36.187 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:36.304 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2020-03-11T22:31:36.735 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:37.118 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:37.118 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:37.118 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2020-03-11T22:31:37.118 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:37.118 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:37.119 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:37.120 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 21 k 2020-03-11T22:31:37.120 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 76 k 2020-03-11T22:31:37.120 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:37.310 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 112 kB/s | 21 kB 00:00 2020-03-11T22:31:37.310 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:37.311 INFO:teuthology.orchestra.run.smithi045.stdout:Total 111 kB/s | 21 kB 00:00 2020-03-11T22:31:37.313 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:37.317 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:37.317 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:37.319 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:37.320 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:37.376 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:37.728 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:37.952 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:38.124 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2020-03-11T22:31:38.552 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:38.835 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:38.947 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:38.948 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:38.948 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:39.106 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2020-03-11T22:31:39.535 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:39.820 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:39.933 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:39.934 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:39.934 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:40.087 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2020-03-11T22:31:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:40.793 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:40.904 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:40.905 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:40.905 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:41.065 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2020-03-11T22:31:41.492 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:41.786 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:41.900 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:41.901 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:41.901 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:42.461 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2020-03-11T22:31:42.946 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:43.228 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:43.340 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:43.341 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:43.341 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:43.472 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2020-03-11T22:31:43.900 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:44.306 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:44.307 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2020-03-11T22:31:44.308 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:44.439 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2020-03-11T22:31:44.868 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:45.254 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:45.255 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:45.255 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:45.255 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:45.257 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 72 k 2020-03-11T22:31:45.257 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 230 k 2020-03-11T22:31:45.257 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:45.452 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 369 kB/s | 72 kB 00:00 2020-03-11T22:31:45.516 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:45.516 INFO:teuthology.orchestra.run.smithi045.stdout:Total 367 kB/s | 72 kB 00:00 2020-03-11T22:31:45.516 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:45.516 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:45.517 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:45.520 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:45.520 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:45.628 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:45.793 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:45.989 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:46.245 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:46.246 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:46.508 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2020-03-11T22:31:46.948 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:47.354 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:47.355 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.7 M 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 12 M 2020-03-11T22:31:47.356 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:48.579 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 2.2 MB/s | 2.7 MB 00:01 2020-03-11T22:31:48.580 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:48.580 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.2 MB/s | 2.7 MB 00:01 2020-03-11T22:31:48.583 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:48.599 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:48.599 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:48.667 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:48.667 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:49.965 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:50.301 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:50.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:50.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:50.560 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:50.560 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:50.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:50.560 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:50.855 INFO:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2020-03-11T22:31:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:55 ago on Wed 11 Mar 2020 10:29:56 PM UTC. 2020-03-11T22:31:51.674 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2020-03-11T22:31:51.674 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:51.674 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2020-03-11T22:31:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2020-03-11T22:31:51.676 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:51.676 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 141 k 2020-03-11T22:31:51.676 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 526 k 2020-03-11T22:31:51.676 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2020-03-11T22:31:52.178 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 281 kB/s | 141 kB 00:00 2020-03-11T22:31:52.178 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:52.179 INFO:teuthology.orchestra.run.smithi045.stdout:Total 280 kB/s | 141 kB 00:00 2020-03-11T22:31:52.180 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2020-03-11T22:31:52.193 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2020-03-11T22:31:52.193 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2020-03-11T22:31:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2020-03-11T22:31:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2020-03-11T22:31:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2020-03-11T22:31:52.542 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:53.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:53.042 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:53.043 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2020-03-11T22:31:53.043 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:53.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2020-03-11T22:31:53.043 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2020-03-11T22:31:53.355 DEBUG:teuthology.parallel:result is None 2020-03-11T22:31:53.355 INFO:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2020-03-11T22:31:53.395 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS 2020-03-11T22:31:53.396 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:53.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:53.631 INFO:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:31:53.650 INFO:teuthology.orchestra.run.smithi045.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:31:53.651 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:31:53.651 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:31:53.651 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:31:53.653 INFO:teuthology.orchestra.run.smithi045:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:31:53.724 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:31:53.725 INFO:teuthology.orchestra.run.smithi045:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:31:53.795 INFO:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:31:53.866 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:31:53.866 INFO:teuthology.orchestra.run.smithi045:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:31:53.951 INFO:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:31:54.021 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:31:54.042 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:54.042 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:31:54.042 INFO:tasks.workunit:timeout=3h 2020-03-11T22:31:54.043 INFO:tasks.workunit:cleanup=True 2020-03-11T22:31:54.043 INFO:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:31:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:31:54.075 INFO:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:54.129 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:54.129 INFO:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:31:54.183 INFO:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:54.278 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:33:30.836 INFO:tasks.workunit.client.0.smithi045.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:33:30.837 INFO:tasks.workunit.client.0.smithi045.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:33:30.838 INFO:tasks.workunit.client.0.smithi045.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:33:30.838 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:33:30.838 INFO:tasks.workunit.client.0.smithi045.stderr: git checkout -b 2020-03-11T22:33:30.838 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:33:30.838 INFO:tasks.workunit.client.0.smithi045.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:33:30.841 INFO:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:33:30.914 DEBUG:teuthology.orchestra.remote:smithi045:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:33:30.962 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:33:30.962 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:33:30.962 INFO:teuthology.orchestra.run.smithi045:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:33:30.992 INFO:tasks.workunit.client.0.smithi045.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:33:31.001 INFO:tasks.workunit.client.0.smithi045.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:33:31.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:33:31.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:33:31.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:33:31.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:33:31.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:33:31.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:33:31.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:33:31.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:33:31.005 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:33:31.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:33:31.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:33:31.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:33:31.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:33:31.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:33:31.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:33:31.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:33:31.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:31.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:31.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:33:31.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:33:31.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none ' 2020-03-11T22:33:31.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:33:31.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:33:31.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:33:31.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:31.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:33:31.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_divergent_2 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_divergent_3 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_divergent_ec' 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:33:31.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:33:31.018 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:33:31.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:33:31.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:33:31.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:33:31.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:33:31.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:33:31.021 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:33:31.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:31.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:33:31.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:33:31.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:31.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:33:31.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:33:31.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:33:31.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:33:31.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:31.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:33:31.026 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:31.027 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:31.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17438 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:33:31.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:33:31.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:33:31.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:33:31.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:33:31.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:33:31.030 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:33:31.031 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:33:31.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:33:31.038 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:33:31.038 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:33:31.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:33:31.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:33:31.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:31.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:31.536 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:31.536 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:31.537 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:31.537 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:31.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:33:31.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:33:31.714 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:33:31.714 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:31.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:31.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:33:31.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:33:31.715 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:31.716 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:31.716 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:31.716 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:31.716 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:31.717 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:31.717 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:33:31.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:31.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get fsid 2020-03-11T22:33:31.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:33:31.811 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:31.811 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:31.811 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:33:31.811 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:33:31.812 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:31.812 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:31.812 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:31.812 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:31.813 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:31.813 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:31.813 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:33:31.813 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:31.814 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get mon_host 2020-03-11T22:33:31.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:33:31.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:33:31.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:33:31.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:33:31.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:33:31.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:33:31.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:33:32.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:33:32.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:32.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:32.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:32.416 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:32.416 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:33:32.416 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:32.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:32.418 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:32.419 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:32.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:32.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:32.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:32.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:33:32.421 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:32.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:33:32.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 8f157d4b-2178-4470-956f-1b1b413295e3' 2020-03-11T22:33:32.422 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:32.422 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:33:32.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA8Z2lePkTOGRAAqZjBlfJBcdj/9y1fiMsW+g== 2020-03-11T22:33:32.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA8Z2lePkTOGRAAqZjBlfJBcdj/9y1fiMsW+g=="}' 2020-03-11T22:33:32.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8f157d4b-2178-4470-956f-1b1b413295e3 -i td/divergent-priors/0/new.json 2020-03-11T22:33:32.905 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:33:32.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:33:32.915 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8Z2lePkTOGRAAqZjBlfJBcdj/9y1fiMsW+g== --osd-uuid 8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:33:32.946 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.941+0000 7f253631dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:32.947 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.943+0000 7f253631dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:32.949 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.945+0000 7f253631dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:32.950 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.946+0000 7f253631dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:32.950 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.946+0000 7f253631dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:32.950 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.946+0000 7f253631dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:32.996 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:32.992+0000 7f253631dec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:33:36.070 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:33:36.070 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:36.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:33:36.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:36.071 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:33:36.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:33:36.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:36.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:36.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:36.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:36.352 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:33:36.391 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:36.387+0000 7f1cd56cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:36.393 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:36.389+0000 7f1cd56cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:36.394 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:36.391+0000 7f1cd56cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:36.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:33:36.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:36.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:36.636 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:33:36.914 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:36.911+0000 7f1cd56cfec0 -1 Falling back to public interface 2020-03-11T22:33:37.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:37.674 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:37.670+0000 7f1cd56cfec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:33:38.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:38.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:38.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:38.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:38.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:38.040 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:33:38.430 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17820,v1:127.0.0.1:6803/17820] [v2:127.0.0.1:6804/17820,v1:127.0.0.1:6805/17820] exists,up 8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:33:38.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:38.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:38.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:38.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:38.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:38.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:33:38.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:38.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:33:38.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 9828c3d9-9600-468d-a840-83ffbcfaf1d5' 2020-03-11T22:33:38.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:38.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBCZ2le1sSoGhAANwCvJW64nvPn13CMUOWLxQ== 2020-03-11T22:33:38.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBCZ2le1sSoGhAANwCvJW64nvPn13CMUOWLxQ=="}' 2020-03-11T22:33:38.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9828c3d9-9600-468d-a840-83ffbcfaf1d5 -i td/divergent-priors/1/new.json 2020-03-11T22:33:38.993 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:33:39.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:33:39.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBCZ2le1sSoGhAANwCvJW64nvPn13CMUOWLxQ== --osd-uuid 9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:33:39.034 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.030+0000 7fa7b2927ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.035 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.032+0000 7fa7b2927ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.036 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.033+0000 7fa7b2927ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.037 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.033+0000 7fa7b2927ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:39.037 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.033+0000 7fa7b2927ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:39.037 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.033+0000 7fa7b2927ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:39.106 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:39.103+0000 7fa7b2927ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:33:42.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:33:42.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:42.200 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:33:42.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:33:42.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:42.683 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:33:42.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:33:42.684 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:42.684 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:42.684 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:42.684 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:42.711 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:42.707+0000 7f0370d43ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.712 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:42.709+0000 7f0370d43ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.714 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:42.710+0000 7f0370d43ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:43.070 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:33:43.070 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:43.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:43.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:33:43.072 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:33:43.229 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:43.225+0000 7f0370d43ec0 -1 Falling back to public interface 2020-03-11T22:33:43.535 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:43.990 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:43.986+0000 7f0370d43ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:33:44.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:44.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:44.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:44.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:44.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:33:44.537 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:33:44.957 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18230,v1:127.0.0.1:6811/18230] [v2:127.0.0.1:6812/18230,v1:127.0.0.1:6813/18230] exists,up 9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:33:44.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:44.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:44.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:44.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:33:44.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:44.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:44.960 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:44.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:44.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:33:44.963 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:44.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:33:44.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 af144a31-aa07-4a4c-b407-e8bbd4765b76' 2020-03-11T22:33:44.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:44.964 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:33:44.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBIZ2leUc8iOhAA42JzF1EG/y6JOnaN8gRkMw== 2020-03-11T22:33:44.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBIZ2leUc8iOhAA42JzF1EG/y6JOnaN8gRkMw=="}' 2020-03-11T22:33:44.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new af144a31-aa07-4a4c-b407-e8bbd4765b76 -i td/divergent-priors/2/new.json 2020-03-11T22:33:45.526 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:33:45.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:33:45.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBIZ2leUc8iOhAA42JzF1EG/y6JOnaN8gRkMw== --osd-uuid af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:33:45.567 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.564+0000 7feab61a2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.615 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.611+0000 7feab61a2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.616 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.612+0000 7feab61a2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.617 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.613+0000 7feab61a2ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:45.617 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.613+0000 7feab61a2ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:45.617 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.613+0000 7feab61a2ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:45.673 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:45.670+0000 7feab61a2ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:33:48.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:33:48.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:48.784 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:33:48.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:33:48.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:49.273 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:33:49.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:33:49.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:49.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:49.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:49.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:49.301 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:49.297+0000 7f4b76df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.303 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:49.299+0000 7f4b76df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.304 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:49.301+0000 7f4b76df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:49.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:49.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:49.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:49.640 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:33:49.820 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:49.816+0000 7f4b76df1ec0 -1 Falling back to public interface 2020-03-11T22:33:50.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:50.650 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:33:50.646+0000 7f4b76df1ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:33:51.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:51.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:51.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:51.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:51.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:51.032 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:33:51.408 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18593,v1:127.0.0.1:6819/18593] [v2:127.0.0.1:6820/18593,v1:127.0.0.1:6821/18593] exists,up af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:33:51.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:51.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:51.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:51.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:33:52.026 INFO:tasks.workunit.client.0.smithi045.stderr:noout is set 2020-03-11T22:33:52.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:33:52.770 INFO:tasks.workunit.client.0.smithi045.stderr:noin is set 2020-03-11T22:33:52.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:33:53.438 INFO:tasks.workunit.client.0.smithi045.stderr:nodown is set 2020-03-11T22:33:53.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:33:53.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:33:54.037 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:33:54.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:33:55.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:33:55.753 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 3 2020-03-11T22:33:55.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:33:56.429 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 min_size to 2 2020-03-11T22:33:56.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:33:56.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:56.438 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:33:56.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:33:56.818 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:33:56.818 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:33:56.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:33:56.819 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:56.819 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:33:56.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:33:56.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:33:56.988 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:56.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:33:57.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:33:57.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:33:57.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:57.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:33:57.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:33:57.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:33:57.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:57.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:33:57.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:57.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:33:57.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:33:57.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:57.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:33:57.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:33:57.265 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:33:57.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:57.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:33:57.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:33:58.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:33:58.634 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:59.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:33:59.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:59.011 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:33:59.011 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:59.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:33:59.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:33:59.014 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:59.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:33:59.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:33:59.015 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:33:59.015 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:59.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:33:59.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:59.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:33:59.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:59.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:33:59.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:33:59.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:59.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:33:59.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:33:59.437 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:33:59.438 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:33:59.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:33:59.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:33:59.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:33:59.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:33:59.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:33:59.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:33:59.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:33:59.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:33:59.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:33:59.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:33:59.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:59.897 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:00.262 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:00.262 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:34:00.262 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:34:00.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:00.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:00.263 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:00.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:34:00.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:34:00.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:00.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:00.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:34:00.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:34:00.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:00.583 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:00.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:34:00.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:34:00.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:00.734 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:34:00.734 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:00.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:00.736 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:34:00.736 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:00.737 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:34:00.737 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:34:00.737 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:34:00.737 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:01.100 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:34:01.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:01.101 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:34:01.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:01.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:01.103 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:34:01.104 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:01.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:34:01.105 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:34:01.105 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:34:01.105 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:01.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672968 2020-03-11T22:34:01.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:02.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:02.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:02.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:34:02.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:02.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:34:02.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:02.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:02.854 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:34:02.854 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:02.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:34:02.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:34:02.855 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:34:02.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:03.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:34:03.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:03.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:03.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:03.738 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:03.738 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:03.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:03.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:03.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:03.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:03.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:03.740 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:04.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:04.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:04.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:04.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:05.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:05.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:05.192 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:05.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:34:05.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:34:05.609 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:34:05.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:34:05.615 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:34:05.615 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:05.615 INFO:tasks.workunit.client.0.smithi045.stdout:primary and soon to be divergent is 1 2020-03-11T22:34:05.973 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:05.974 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:33:54.850686+0000 0'0 27:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:34:05.974 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:34:05.974 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:05.975 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:34:05.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:34:05.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:34:05.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:34:05.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:34:05.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:34:05.986 INFO:tasks.workunit.client.0.smithi045.stdout:writing initial objects 2020-03-11T22:34:05.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:05.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:34:06.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:34:06.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:34:06.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:34:06.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:34:06.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:34:06.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:34:06.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:34:06.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:34:06.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:34:06.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:34:06.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:34:06.693 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.693 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:34:06.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:34:06.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:34:06.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:06.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:34:07.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.070 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:34:07.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:34:07.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:34:07.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:34:07.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:34:07.286 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.286 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:34:07.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:34:07.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:34:07.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:34:07.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:34:07.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:34:07.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:34:07.837 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.837 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:34:07.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:34:07.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:34:07.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:07.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:34:08.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:34:08.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:34:08.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:34:08.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:34:08.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:34:08.388 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.388 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:34:08.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:34:08.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:34:08.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:34:08.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:34:08.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:34:08.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.837 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:34:08.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:34:08.920 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.920 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:34:08.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:08.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:34:09.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:34:09.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:34:09.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:34:09.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:34:09.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:34:09.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:34:09.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:34:09.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:34:09.593 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.593 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:34:09.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:34:09.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:34:09.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:34:09.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:34:09.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:34:09.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:34:09.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:09.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:34:10.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:34:10.120 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.120 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:34:10.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:34:10.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:34:10.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:34:10.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:34:10.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:34:10.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:34:10.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:34:10.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:34:10.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:34:10.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:34:10.834 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.834 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:34:10.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:34:10.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:10.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:34:11.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:34:11.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:34:11.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:34:11.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:34:11.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:34:11.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:34:11.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:34:11.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:34:11.619 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.619 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:34:11.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:34:11.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:34:11.811 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.811 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:34:11.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:34:11.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:34:11.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:11.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:34:12.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:34:12.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:34:12.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:34:12.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:34:12.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:34:12.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:34:12.487 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:12.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:12.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:34:12.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:12.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:12.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:12.539 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:12.539 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:12.540 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:12.540 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:12.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:12.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:12.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:12.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:12.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:12.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:12.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:12.597 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:12.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:12.959 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:34:12.959 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:34:12.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:12.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:12.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:13.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:34:13.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:34:13.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:13.107 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:13.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:34:13.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:34:13.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:13.244 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:13.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:34:13.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:34:13.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:13.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:34:13.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:13.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:13.382 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:34:13.382 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:13.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:34:13.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:34:13.383 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:34:13.384 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:13.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:34:13.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:14.746 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:14.747 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:15.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836493 2020-03-11T22:34:15.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:15.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:15.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:15.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:15.126 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:15.126 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:15.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:34:15.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:34:15.128 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:34:15.128 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:15.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:34:15.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:15.492 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:34:15.492 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:15.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:15.494 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:34:15.494 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:15.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:34:15.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:34:15.495 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:34:15.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:15.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:34:15.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:15.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:15.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:16.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:16.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:16.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:16.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:16.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:16.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:16.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:16.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:16.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:16.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:16.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:16.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:17.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:17.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:17.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:17.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:34:17.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:34:17.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:34:17.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:34:17.175 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:34:17.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:34:17.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:34:17.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:17.550 INFO:tasks.workunit.client.0.smithi045.stdout:blackholing osds 0 2 2020-03-11T22:34:17.935 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:17.935 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:33:54.850686+0000 28'100 28:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:34:17.936 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:34:17.936 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:17.937 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:34:17.945 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:17.946 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:34:17.946 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:34:17.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:34:17.947 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:17.947 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:17.947 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:17.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:34:17.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:17.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:34:18.038 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:34:18.038 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:18.038 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:34:18.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:18.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:34:18.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:34:18.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:34:18.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:18.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:18.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:18.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:34:18.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:18.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:34:18.139 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:34:18.139 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:18.139 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:34:18.146 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:34:18.147 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:34:18.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:34:18.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:34:18.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stdout:writing divergent object 2020-03-11T22:34:18.153 INFO:tasks.workunit.client.0.smithi045.stdout:create missing divergent object 2020-03-11T22:34:18.518 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:34:18.518 INFO:tasks.workunit.client.0.smithi045.stderr:0 2020-03-11T22:34:18.518 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:34:18.518 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:34:18.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:34:18.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:34:18.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:34:18.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:34:18.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:34:18.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:34:18.522 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:34:18.522 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:34:18.522 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:18.522 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:18.522 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:18.523 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:18.523 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:18.523 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:34:18.523 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17438/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:34:18.621 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:34:18.621 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:34:18.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:34:18.623 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:34:18.623 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:18.624 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:18.627 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:18.628 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:18.628 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:18.628 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:34:18.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:34:18.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:34:18.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:34:18.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:34:28.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:34:28.745 INFO:tasks.workunit.client.0.smithi045.stderr:rados: no process found 2020-03-11T22:34:28.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:34:28.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:28.745 INFO:tasks.workunit.client.0.smithi045.stdout:killing all the osds 2020-03-11T22:34:29.099 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:29.099 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:34:19.007108+0000 28'101 28:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:34:29.099 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:34:29.099 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:29.100 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:34:29.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:34:29.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:34:29.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:34:29.110 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:34:29.110 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:34:29.110 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:34:29.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:34:29.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:29.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:34:29.728 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already down. 2020-03-11T22:34:29.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:29.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:34:30.247 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already down. 2020-03-11T22:34:30.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:30.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:34:30.809 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already down. 2020-03-11T22:34:30.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:30.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:34:31.382 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already out. 2020-03-11T22:34:31.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:31.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:34:31.911 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already out. 2020-03-11T22:34:31.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:34:31.920 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:31.920 INFO:tasks.workunit.client.0.smithi045.stdout:bringing up non_divergent 0 2 2020-03-11T22:34:32.257 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:32.257 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:34:19.007108+0000 28'101 28:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:34:32.258 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:34:32.258 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:32.259 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:34:32.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:32.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:34:32.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:32.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:32.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:32.273 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:32.275 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:34:32.276 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:34:32.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:34:32.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:32.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:32.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:32.298 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:32.294+0000 7fe4e7a49ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:32.300 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:32.296+0000 7fe4e7a49ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:32.301 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:32.297+0000 7fe4e7a49ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:32.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:32.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:32.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:32.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:32.682 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:34:32.822 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:32.818+0000 7fe4e7a49ec0 -1 Falling back to public interface 2020-03-11T22:34:33.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:33.755 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:33.752+0000 7fe4e7a49ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:34:34.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:34.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:34.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:34.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:34.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:34.054 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:34:34.430 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22570,v1:127.0.0.1:6803/22570] [v2:127.0.0.1:6804/22570,v1:127.0.0.1:6805/22570] exists,up 8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:34.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:34.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:34.433 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:34:34.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:34.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:34.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:34.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:34.457 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:34.453+0000 7f166366bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:34.458 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:34.455+0000 7f166366bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:34.460 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:34.456+0000 7f166366bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:34.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:34:34.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:34.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:34:34.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:34.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:34.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:34.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:34.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:34.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:34.800 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:34:34.995 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:34.992+0000 7f166366bec0 -1 Falling back to public interface 2020-03-11T22:34:35.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:35.905 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:34:35.902+0000 7f166366bec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:34:36.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:36.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:36.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:36.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:36.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:36.191 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:34:36.760 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22764,v1:127.0.0.1:6811/22764] [v2:127.0.0.1:6812/22764,v1:127.0.0.1:6813/22764] exists,up af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:34:36.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:36.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:36.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:36.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:36.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:34:37.324 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already in. 2020-03-11T22:34:37.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:37.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:34:37.934 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already in. 2020-03-11T22:34:37.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:37.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:34:37.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:37.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:37.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:37.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:37.943 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:37.944 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:37.944 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:37.944 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:37.944 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:37.944 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:38.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:38.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:38.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:38.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:38.001 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:38.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:38.402 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:34:38.403 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:34:38.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:38.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:38.403 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:38.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:34:38.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:34:38.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:38.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:38.602 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:34:38.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:34:38.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:34:38.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:38.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:38.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:34:38.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:34:38.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:38.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:34:38.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:38.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:38.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:34:38.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:38.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:34:38.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:34:38.750 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:38.751 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:34:39.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:34:39.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:39.113 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:34:39.113 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:39.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:39.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:34:39.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:39.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:34:39.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:34:39.116 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:34:39.116 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:39.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:34:39.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:34:39.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:39.504 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:34:39.504 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:39.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:39.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:34:39.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:39.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:34:39.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:34:39.506 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:39.507 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:34:39.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:34:39.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:40.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:40.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:41.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:34:41.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:41.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:41.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:41.748 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:41.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:41.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:41.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:41.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:41.750 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:41.750 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:41.750 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:42.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:42.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:42.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:34:42.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:34:42.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:34:42.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:42.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:34:42.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:42.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:42.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:43.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:43.042 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:43.042 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:43.042 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:43.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:43.527 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:43.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:43.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:43.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:43.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:34:43.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:43.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:34:44.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:44.199 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:44.200 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:44.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:44.552 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:44.552 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:44.552 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:45.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:45.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:45.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:45.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:45.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:45.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:45.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:34:45.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:45.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:34:45.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:45.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:45.897 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:45.897 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:45.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:45.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:45.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:45.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:46.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:46.255 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:46.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:46.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:46.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:46.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:46.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:46.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:47.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:47.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:47.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:34:47.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:47.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:34:48.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:48.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:48.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:48.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:48.385 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:48.385 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:48.385 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:48.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:48.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:48.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:48.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:48.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:48.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:48.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:48.861 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:48.861 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:49.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:49.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:49.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:34:49.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:49.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:34:50.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:50.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:50.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:50.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:50.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:50.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:50.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:50.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:51.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:51.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:51.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:51.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:51.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:51.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:51.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:51.770 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:52.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:34:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:34:55.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:55.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:55.441 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:55.441 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:55.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:55.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:55.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:55.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:55.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:55.805 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:55.806 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:55.806 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:56.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:56.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:56.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:56.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:56.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:56.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:56.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:56.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:56.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:56.920 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:56.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:56.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:34:56.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:56.921 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:35:03.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:03.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:03.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:03.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:03.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:03.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:03.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:03.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:03.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:03.684 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:03.684 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:03.684 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:04.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:04.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:04.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:04.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:04.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:04.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:04.627 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:35:04.627 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:04.627 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:35:11.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:11.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:11.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:11.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:11.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:11.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:11.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:11.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:12.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:12.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:12.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:12.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:12.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:12.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:12.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:12.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:13.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:13.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:13.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:35:13.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:35:13.700 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:35:13.700 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "e3dfbcfa-e2e8-4308-b7d4-957eb601da7a", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:35:13.697020+0000", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:35:13.701 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.702 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:35:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:35:13.704 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:35:13.706 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:35:13.707 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:35:13.709 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:33:31.089654Z", 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:33:31.089654Z", 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:35:13.710 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:35:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:35:13.712 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.714 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 38, 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:35:13.715 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:33:31.981095+0000", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:34:37.820258+0000", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:34:35.977681+0000", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:34:37.643309+0000", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:35:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 7, 2020-03-11T22:35:13.717 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:35:13.718 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:33:53.718804+0000", 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 1, 2020-03-11T22:35:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool", 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 0, 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:35:13.720 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 1, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 1, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 1, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 1, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:35:13.721 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "28", 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:35:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 0, 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 0, 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [], 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:35:13.723 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:35:13.724 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "", 2020-03-11T22:35:13.725 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:35:13.726 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 0, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:35:13.727 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "8f157d4b-2178-4470-956f-1b1b413295e3", 2020-03-11T22:35:13.728 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 28, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 34, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 37, 2020-03-11T22:35:13.729 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 29, 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:35:13.730 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.731 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.732 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.733 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.734 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:13.735 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.736 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22570 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/22570", 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/22570", 2020-03-11T22:35:13.737 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22570", 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22570", 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.738 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "9828c3d9-9600-468d-a840-83ffbcfaf1d5", 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:35:13.739 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 22, 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 30, 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.740 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:13.741 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.742 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.743 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:13.744 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.745 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.746 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 18230 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.747 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/18230", 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/18230", 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18230", 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18230", 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "af144a31-aa07-4a4c-b407-e8bbd4765b76", 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:35:13.749 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 28, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 35, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 31, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.750 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.751 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:35:13.752 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.753 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.754 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:13.755 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:35:13.756 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 22764 2020-03-11T22:35:13.757 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/22764", 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/22764", 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22764", 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22764", 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:35:13.758 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.759 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:34:29.525989+0000", 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:38.628603+0000", 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:35:13.760 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:34:30.048654+0000", 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:35:13.761 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:44.991381+0000", 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:34:30.583276+0000", 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:35:13.762 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:51.684535+0000", 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.763 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:35:13.764 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:35:13.765 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:35:13.766 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/22570,v1:127.0.0.1:6805/22570]", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:13.767 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:13.768 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:35:13.769 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/22570,v1:127.0.0.1:6803/22570]", 2020-03-11T22:35:13.770 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22570,v1:127.0.0.1:6809/22570]", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22570,v1:127.0.0.1:6807/22570]", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:13.771 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:35:13.772 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/18230,v1:127.0.0.1:6813/18230]", 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:35:13.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:35:13.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:13.775 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:13.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/18230,v1:127.0.0.1:6811/18230]", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18230,v1:127.0.0.1:6817/18230]", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18230,v1:127.0.0.1:6815/18230]", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:13.777 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:35:13.778 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/22764,v1:127.0.0.1:6813/22764]", 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:35:13.779 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:13.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:13.781 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:35:13.782 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/22764,v1:127.0.0.1:6811/22764]", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22764,v1:127.0.0.1:6817/22764]", 2020-03-11T22:35:13.783 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22764,v1:127.0.0.1:6815/22764]", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:35:13.784 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.785 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 38, 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:35:13.786 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.787 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:35:13.788 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:35:13.789 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:35:13.790 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.790 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.790 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:35:13.790 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:35:13.790 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:35:13.791 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:35:13.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:35:13.793 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:35:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:35:13.795 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:35:13.796 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:35:13.797 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.798 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:35:13.799 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:35:13.800 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.801 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.802 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:35:13.803 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.804 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:35:13.805 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:35:13.806 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:35:13.807 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.808 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.809 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:35:13.810 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:35:13.811 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 0, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:35:13.812 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:35:13.813 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:35:13.814 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.815 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.816 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:35:13.817 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 1, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:35:13.818 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:13.819 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:13.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:35:13.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:13.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:35:13.823 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:13.824 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 16, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:35:13.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:35:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3178880, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 32896, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 13, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145714, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393920, 2020-03-11T22:35:13.827 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867374080, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33685504, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 397826, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:35:13.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 13730, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221211742 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:13.829 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:13.830 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.831 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:13.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3178880, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 32896, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 13, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145714, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393920, 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:35:13.833 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867374080, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33685504, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 397826, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:13.834 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 13730, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221211742 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:13.835 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:13.836 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:35:13.837 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:35:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:35:13.839 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:13.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:35:13.841 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:13.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:35:13.843 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19660800, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 144900, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:35:13.844 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 16, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:35:13.845 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 146028888074 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672975 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 150323855370 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.847 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 1 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.849 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:35:13.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 1, 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:35:13.851 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:35:13.852 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:33:31.981405+0000", 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 311, 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:35:13.853 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:35:13.854 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:35:13.854 INFO:tasks.workunit.client.0.smithi045.stderr:report 1931777302 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:13.855 INFO:tasks.workunit.client.0.smithi045.stdout:writing non-divergent object existing_4 2020-03-11T22:35:14.056 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:14.056 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 28'100 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:14.057 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:14.057 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:14.058 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:14.066 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:35:14.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:35:14.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:14.204 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery 2020-03-11T22:35:14.546 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:14.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 28'100 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:14.547 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:14.547 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:14.548 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:14.556 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:14.557 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:35:14.557 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:14.558 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:35:14.654 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:14.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:14.663 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:14.664 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:35:14.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:14.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:35:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:14.766 INFO:tasks.workunit.client.0.smithi045.stdout:revive divergent 1 2020-03-11T22:35:14.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:35:14.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:15.111 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:15.111 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 28'100 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:15.111 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:15.112 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:15.113 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:15.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:35:15.764 INFO:tasks.workunit.client.0.smithi045.stderr:noup is set 2020-03-11T22:35:15.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:15.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:15.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:15.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:15.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:15.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:15.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:15.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:15.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:15.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:15.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:15.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:15.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:15.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:15.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:15.782 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:15.782 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:15.782 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:15.783 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:15.783 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:15.783 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:15.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:15.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:15.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:15.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:15.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:15.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:15.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:15.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:15.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:15.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:15.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:15.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:15.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:15.787 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:15.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:15.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:15.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:15.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:15.788 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:35:15.804 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:15.801+0000 7f4f6afe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.805 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:15.803+0000 7f4f6afe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.807 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:15.805+0000 7f4f6afe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:16.147 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:35:16.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:35:16.323 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:16.320+0000 7f4f6afe5ec0 -1 Falling back to public interface 2020-03-11T22:35:17.207 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:17.205+0000 7f4f6afe5ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery divergent 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:21.499 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:21.499 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 38'101 38:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:21.500 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:21.500 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:21.501 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:21.510 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:35:21.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:21.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:35:21.606 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:21.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:35:22.393 INFO:tasks.workunit.client.0.smithi045.stderr:noup is unset 2020-03-11T22:35:22.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:22.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:22.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:22.403 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:35:22.844 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22570,v1:127.0.0.1:6803/22570] [v2:127.0.0.1:6804/22570,v1:127.0.0.1:6805/22570] exists,up 8f157d4b-2178-4470-956f-1b1b413295e3 2020-03-11T22:35:22.844 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:22.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:22.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:23.208 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24235,v1:127.0.0.1:6819/24235] [v2:127.0.0.1:6820/24235,v1:127.0.0.1:6821/24235] exists,up 9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:35:23.208 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:35:23.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:23.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:23.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:23.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:23.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:23.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:23.601 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22764,v1:127.0.0.1:6811/22764] [v2:127.0.0.1:6812/22764,v1:127.0.0.1:6813/22764] exists,up af144a31-aa07-4a4c-b407-e8bbd4765b76 2020-03-11T22:35:23.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:23.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:23.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:23.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:23.944 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:23.944 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 38'101 38:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:23.945 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:23.945 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:23.946 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:23.954 INFO:tasks.workunit.client.0.smithi045.stdout:wait for peering 2020-03-11T22:35:23.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:35:23.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:24.296 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:24.296 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:34:37.963399+0000 38'101 38:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:24.296 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:24.297 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:24.298 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:24.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:35:34.005 INFO:tasks.workunit.client.0.smithi045.stdout:killing divergent 1 2020-03-11T22:35:34.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:35:34.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:34.363 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:34.364 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:22.750200+0000 38'101 44:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:34.364 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:34.364 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:34.365 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:34.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:35:34.374 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:34.374 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:34.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:34.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:34.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:34.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:34.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:35:34.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:34.480 INFO:tasks.workunit.client.0.smithi045.stdout:reviving divergent 1 2020-03-11T22:35:34.842 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:34.843 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:22.750200+0000 38'101 44:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:34.843 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:34.844 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:34.844 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:34.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:34.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:34.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:34.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:34.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:34.860 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:35:34.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:34.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:34.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:34.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:34.883 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:34.879+0000 7f81ecbcaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:34.885 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:34.881+0000 7f81ecbcaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:34.886 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:34.882+0000 7f81ecbcaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:35.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:35:35.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:35.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:35.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:35.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:35.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:35.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:35.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:35.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:35.227 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:35:35.401 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:35.397+0000 7f81ecbcaec0 -1 Falling back to public interface 2020-03-11T22:35:35.597 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24235,v1:127.0.0.1:6819/24235] [v2:127.0.0.1:6820/24235,v1:127.0.0.1:6821/24235] exists,up 9828c3d9-9600-468d-a840-83ffbcfaf1d5 2020-03-11T22:35:35.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:35.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:35.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:35.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:35:36.310 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:36.301+0000 7f81ecbcaec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:35:55.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:35:55.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:55.598 INFO:tasks.workunit.client.0.smithi045.stdout:allowing recovery 2020-03-11T22:35:55.949 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:55.949 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:37.908265+0000 44'102 47:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:55.950 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:55.951 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:55.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:55.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:35:56.087 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:56.094 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:56.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:35:56.222 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:56.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:56.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:35:56.357 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:56.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:35:56.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:56.365 INFO:tasks.workunit.client.0.smithi045.stdout:reading divergent objects 2020-03-11T22:35:56.711 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:56.711 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:37.908265+0000 44'102 47:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:53.777195+0000 0'0 2020-03-11T22:33:53.777195+0000 0 2020-03-11T22:35:56.711 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:35:56.711 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:56.713 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:35:56.721 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:35:56.722 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:35:56.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:56.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:35:56.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:56.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:35:56.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:56.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:35:56.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:56.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:35:56.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:35:56.847 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:35:56.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:35:56.851 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:18.173602+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:18.173602+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-03-11T22:35:56.852 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:18.173602+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:56.853 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:22.642+0000 7f4f491a0700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:35:56.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:35:56.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:35:56.856 INFO:tasks.workunit.client.0.smithi045.stdout:success 2020-03-11T22:35:56.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:35:56.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:35:56.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:35:56.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:35:57.336 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' does not exist 2020-03-11T22:35:57.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:35:57.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:57.344 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:57.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:57.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:57.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:57.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:57.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:35:57.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:57.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:57.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:57.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:57.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:57.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:57.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:57.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:57.457 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:57.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:57.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:57.458 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:57.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:57.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:57.459 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:57.461 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:57.461 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:57.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:57.462 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:57.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:57.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:57.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:57.474 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:57.474 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:57.474 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:57.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:35:57.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:57.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:57.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:57.477 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:57.477 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:57.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:57.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:57.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:57.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:57.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:57.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:57.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:57.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:57.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:57.480 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:57.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:57.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:57.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:57.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:57.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:57.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:57.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:57.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:57.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:57.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:57.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:35:57.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:57.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:57.487 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:35:57.487 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:35:57.487 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:57.487 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:57.487 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17438 2020-03-11T22:35:57.488 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:35:57.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:35:57.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:35:57.490 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:35:57.491 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:35:57.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:35:57.491 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:35:57.491 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:35:57.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:57.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:57.811 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:35:57.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:35:57.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:35:57.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:57.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:57.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:35:57.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:35:57.989 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:57.989 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:57.989 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:57.989 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:57.989 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:57.990 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:57.990 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:35:57.990 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:57.990 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get fsid 2020-03-11T22:35:58.081 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:58.082 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:58.083 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:58.084 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:58.084 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:58.084 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:35:58.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:58.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get mon_host 2020-03-11T22:35:58.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:35:58.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:35:58.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:35:58.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:35:58.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:35:58.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:35:58.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:35:59.091 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:59.092 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:59.093 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:59.093 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:59.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:35:59.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:35:59.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:59.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:59.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:59.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:59.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:59.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:59.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:59.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:59.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:35:59.128 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:59.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:35:59.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 89d0c445-e220-4c12-9579-4ad84b06fa4c' 2020-03-11T22:35:59.129 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:59.129 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:35:59.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDPZ2leQZ5aCBAAmzPNhGe9+CF5cNrtTF/OKA== 2020-03-11T22:35:59.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDPZ2leQZ5aCBAAmzPNhGe9+CF5cNrtTF/OKA=="}' 2020-03-11T22:35:59.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 89d0c445-e220-4c12-9579-4ad84b06fa4c -i td/divergent-priors/0/new.json 2020-03-11T22:35:59.479 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:35:59.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:35:59.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPZ2leQZ5aCBAAmzPNhGe9+CF5cNrtTF/OKA== --osd-uuid 89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:35:59.519 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.515+0000 7faa2aa4bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:59.520 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.517+0000 7faa2aa4bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:59.522 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.520+0000 7faa2aa4bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:59.523 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.520+0000 7faa2aa4bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:59.523 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.520+0000 7faa2aa4bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:59.523 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.520+0000 7faa2aa4bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:35:59.608+0000 7faa2aa4bec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:36:02.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:36:02.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:02.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:36:02.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:02.650 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:03.139 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:36:03.166 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:03.164+0000 7fe593c12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:03.168 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:03.165+0000 7fe593c12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:03.170 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:03.167+0000 7fe593c12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:03.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:36:03.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:03.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:03.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:03.507 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:36:03.685 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:03.682+0000 7fe593c12ec0 -1 Falling back to public interface 2020-03-11T22:36:03.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:04.454 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:04.452+0000 7fe593c12ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:04.872 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:36:05.268 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25537,v1:127.0.0.1:6803/25537] [v2:127.0.0.1:6804/25537,v1:127.0.0.1:6805/25537] exists,up 89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:36:05.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:05.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:05.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:05.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:36:05.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:05.271 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:05.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:05.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:36:05.274 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:05.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:36:05.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8' 2020-03-11T22:36:05.275 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:05.275 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:36:05.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDVZ2leO7z3EBAACf1CDo+9XI832DNwEF/dIg== 2020-03-11T22:36:05.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDVZ2leO7z3EBAACf1CDo+9XI832DNwEF/dIg=="}' 2020-03-11T22:36:05.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 -i td/divergent-priors/1/new.json 2020-03-11T22:36:05.876 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:36:05.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:36:05.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVZ2leO7z3EBAACf1CDo+9XI832DNwEF/dIg== --osd-uuid 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:36:05.916 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.914+0000 7ff4c347fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:05.918 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.915+0000 7ff4c347fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:05.920 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.917+0000 7ff4c347fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:05.921 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.918+0000 7ff4c347fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:05.921 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.918+0000 7ff4c347fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:05.921 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.918+0000 7ff4c347fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:05.952 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:05.950+0000 7ff4c347fec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:36:09.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:36:09.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:09.088 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:36:09.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:36:09.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:09.573 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:36:09.574 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:09.574 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:09.574 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:09.574 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:09.574 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:36:09.599 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:09.597+0000 7f8b7245fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:09.601 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:09.599+0000 7f8b7245fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:09.602 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:09.600+0000 7f8b7245fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:09.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:36:09.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:09.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:09.935 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:36:10.118 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:10.116+0000 7f8b7245fec0 -1 Falling back to public interface 2020-03-11T22:36:10.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:10.877 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:10.875+0000 7f8b7245fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:36:11.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:11.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:11.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:11.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:11.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:11.328 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:36:11.712 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25892,v1:127.0.0.1:6811/25892] [v2:127.0.0.1:6812/25892,v1:127.0.0.1:6813/25892] exists,up 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:36:11.712 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:36:11.712 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:11.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:11.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:11.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:11.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:36:11.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:11.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:11.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:36:11.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:11.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:11.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:11.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:11.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:11.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:11.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:11.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:11.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:11.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:11.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:11.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:11.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:11.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:11.718 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:11.718 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:11.719 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:11.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:11.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:11.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:11.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:11.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:11.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:11.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:11.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:11.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:11.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:11.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:11.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:11.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:11.722 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:11.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:36:11.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d5a42e5b-0e5d-4d4e-b380-83acd1693bcc' 2020-03-11T22:36:11.723 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:11.727 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDbZ2lealAyKxAA5llSK8ch+NJrW0CL9DJxyw== 2020-03-11T22:36:11.727 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDbZ2lealAyKxAA5llSK8ch+NJrW0CL9DJxyw=="}' 2020-03-11T22:36:11.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d5a42e5b-0e5d-4d4e-b380-83acd1693bcc -i td/divergent-priors/2/new.json 2020-03-11T22:36:12.240 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:36:12.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:36:12.251 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDbZ2lealAyKxAA5llSK8ch+NJrW0CL9DJxyw== --osd-uuid d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:36:12.280 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.278+0000 7f2e54e44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:12.282 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.279+0000 7f2e54e44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:12.284 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.281+0000 7f2e54e44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:12.284 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.282+0000 7f2e54e44ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:12.284 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.282+0000 7f2e54e44ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:12.285 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.282+0000 7f2e54e44ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:12.335 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:12.332+0000 7f2e54e44ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:36:15.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:36:15.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:15.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:36:15.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:15.500 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:36:15.985 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:36:15.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:36:15.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:15.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:15.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:15.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:16.012 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:16.010+0000 7fe5448d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:16.014 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:16.012+0000 7fe5448d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:16.016 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:16.014+0000 7fe5448d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:16.362 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:36:16.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:16.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:16.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:16.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:16.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:16.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:16.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:16.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:16.364 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:36:16.536 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:16.533+0000 7fe5448d9ec0 -1 Falling back to public interface 2020-03-11T22:36:16.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:17.319 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:17.317+0000 7fe5448d9ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:36:17.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:17.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:17.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:17.759 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:36:17.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:17.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:18.137 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26247,v1:127.0.0.1:6819/26247] [v2:127.0.0.1:6820/26247,v1:127.0.0.1:6821/26247] exists,up d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:36:18.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:18.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:18.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:18.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:36:18.863 INFO:tasks.workunit.client.0.smithi045.stderr:noout is set 2020-03-11T22:36:18.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:36:19.599 INFO:tasks.workunit.client.0.smithi045.stderr:noin is set 2020-03-11T22:36:19.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:36:20.325 INFO:tasks.workunit.client.0.smithi045.stderr:nodown is set 2020-03-11T22:36:20.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:36:20.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:36:20.878 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:36:20.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:36:21.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:36:22.555 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 3 2020-03-11T22:36:22.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:36:23.223 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 min_size to 2 2020-03-11T22:36:23.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:36:23.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:23.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:23.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:23.596 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:36:23.597 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:36:23.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:23.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:23.597 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:23.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:36:23.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:36:23.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:23.734 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:23.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:36:23.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:36:23.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:23.870 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:24.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:24.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:24.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:24.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:36:24.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:36:24.011 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:24.011 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:36:24.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:36:24.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:25.385 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:25.386 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:25.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:36:25.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:25.761 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:36:25.761 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:25.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:25.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:36:25.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:25.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:36:25.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:36:25.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:25.764 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:36:26.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:36:26.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:26.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:26.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:26.169 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:26.170 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:26.170 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:26.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:36:26.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:36:26.171 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:26.171 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:36:26.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:36:26.560 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:26.561 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:26.561 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:26.561 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:26.561 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:26.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:26.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:26.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:26.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:26.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:27.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:27.021 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:36:27.021 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:36:27.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:27.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:27.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:27.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:36:27.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:36:27.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:27.157 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:27.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:36:27.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:36:27.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:27.295 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:36:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:36:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:27.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:27.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:27.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:27.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:27.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:27.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:36:27.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:36:27.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:27.447 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:36:27.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:36:27.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:28.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:28.814 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:29.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:36:29.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:29.189 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:29.189 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:29.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:29.190 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:29.190 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:29.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:36:29.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:36:29.191 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:36:29.191 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:29.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:36:29.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:29.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:36:29.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:29.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:29.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:36:29.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:29.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:36:29.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:36:29.592 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:36:29.592 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:30.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:36:30.073 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:30.074 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:30.074 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:30.580 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:30.580 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:30.580 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:30.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:30.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:30.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:30.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:30.582 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:30.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:30.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:30.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:30.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:31.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:31.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:31.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:31.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:36:31.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:36:31.750 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:36:31.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:36:31.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:36:31.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:31.758 INFO:tasks.workunit.client.0.smithi045.stdout:primary and soon to be divergent is 1 2020-03-11T22:36:32.101 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:32.101 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:36:21.618035+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:36:32.101 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:36:32.101 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:32.102 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:36:32.110 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:36:32.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:32.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:32.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:36:32.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:36:32.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:36:32.113 INFO:tasks.workunit.client.0.smithi045.stdout:writing initial objects 2020-03-11T22:36:32.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:36:32.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:36:32.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:36:32.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:36:32.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:36:32.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:36:32.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:36:32.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:36:32.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:36:32.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:36:32.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:36:32.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:36:32.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:36:32.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:36:32.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:36:32.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:32.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:36:33.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:36:33.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:36:33.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:36:33.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:36:33.309 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:36:33.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:36:33.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:36:33.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:36:33.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:36:33.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:36:33.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:36:33.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:36:33.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:36:33.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:36:33.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:36:33.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:33.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:36:34.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:36:34.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:36:34.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:36:34.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:36:34.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:36:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:36:34.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.385 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:36:34.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:36:34.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:36:34.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:36:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:36:34.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:36:34.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:36:34.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:36:34.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:34.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:36:35.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:36:35.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.110 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:36:35.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:36:35.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:36:35.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:36:35.302 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.302 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:36:35.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:36:35.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:36:35.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:36:35.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:36:35.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.619 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:36:35.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:36:35.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:36:35.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:36:35.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:36:35.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.911 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:36:35.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:35.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:36:36.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:36:36.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:36:36.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:36:36.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:36:36.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:36:36.378 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.378 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:36:36.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:36:36.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:36:36.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:36:36.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.646 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:36:36.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:36:36.752 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.752 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:36:36.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:36:36.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:36:36.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:36:36.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:36.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:36:37.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:36:37.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:36:37.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:36:37.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:36:37.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:36:37.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:36:37.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:36:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:36:37.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:36:37.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:36:37.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:36:37.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:36:37.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:36:37.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:36:37.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:37.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:36:38.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:36:38.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:36:38.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:36:38.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:36:38.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:36:38.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:38.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:36:38.457 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:38.458 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:38.459 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:38.459 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:38.459 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:38.460 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:38.460 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:38.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:38.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:38.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:38.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:38.518 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:38.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:38.882 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:36:38.883 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:36:38.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:38.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:38.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:39.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:36:39.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:36:39.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:39.026 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:39.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:36:39.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:36:39.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:39.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:39.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:36:39.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:36:39.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:39.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:39.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:39.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:39.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:39.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:39.301 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:36:39.301 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:36:39.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:39.301 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:36:39.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:36:39.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:40.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:40.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:41.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:36:41.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:41.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:41.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:41.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:41.044 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:41.045 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:41.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:36:41.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:36:41.046 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:36:41.046 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:41.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:36:41.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:41.422 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:41.422 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:41.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:41.423 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:41.423 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:41.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:36:41.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:36:41.424 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:41.425 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:36:41.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:36:41.794 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:41.794 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:41.794 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:42.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:42.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:42.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:42.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:42.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:42.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:42.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:42.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:42.629 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:42.629 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:42.629 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:42.630 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:43.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:43.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:43.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:43.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:36:43.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:36:43.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:36:43.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:36:43.102 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:36:43.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:36:43.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:36:43.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:43.471 INFO:tasks.workunit.client.0.smithi045.stdout:blackholing osds 0 2 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:36:21.618035+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:36:43.814 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:36:44.016 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:36:44.016 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:44.016 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:36:44.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:36:44.023 INFO:tasks.workunit.client.0.smithi045.stdout:create new divergent objects 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:36:44.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:36:44.026 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:36:44.026 INFO:tasks.workunit.client.0.smithi045.stdout:writing divergent objects 2020-03-11T22:36:44.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:44.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:44.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:36:44.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:36:44.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:36:44.028 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:36:44.028 INFO:tasks.workunit.client.0.smithi045.stdout:remove divergent objects 2020-03-11T22:36:44.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:44.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:36:44.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:36:44.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:44.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:36:44.030 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:36:44.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:36:44.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:36:44.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:36:54.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:36:54.037 INFO:tasks.workunit.client.0.smithi045.stderr:rados: no process found 2020-03-11T22:36:54.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:36:54.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:54.037 INFO:tasks.workunit.client.0.smithi045.stdout:killing all the osds 2020-03-11T22:36:54.390 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:54.390 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:36:21.618035+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:36:54.390 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:36:54.390 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:54.392 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:36:54.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:36:54.400 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:54.400 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:54.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:54.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:54.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:54.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:54.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:54.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:36:55.065 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already down. 2020-03-11T22:36:55.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:55.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:36:55.592 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already down. 2020-03-11T22:36:55.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:55.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:36:56.108 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already down. 2020-03-11T22:36:56.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:56.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already out. 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:36:57.163 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already out. 2020-03-11T22:36:57.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:36:57.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:57.172 INFO:tasks.workunit.client.0.smithi045.stdout:bringing up non_divergent 0 2 2020-03-11T22:36:57.510 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:57.510 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:36:21.618035+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:36:57.510 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:36:57.511 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:57.512 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:36:57.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:57.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:57.525 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:57.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:36:57.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:57.528 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:36:57.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:36:57.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:36:57.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:57.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:57.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:57.551 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:57.547+0000 7f3166c89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:57.552 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:57.549+0000 7f3166c89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:57.554 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:57.550+0000 7f3166c89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:57.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:36:57.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:57.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:57.892 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:36:58.078 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:58.075+0000 7f3166c89ec0 -1 Falling back to public interface 2020-03-11T22:36:58.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:58.912 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:58.909+0000 7f3166c89ec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:36:59.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:59.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:59.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:59.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:59.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:59.259 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:36:59.638 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/30297,v1:127.0.0.1:6803/30297] [v2:127.0.0.1:6804/30297,v1:127.0.0.1:6805/30297] exists,up 89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:36:59.638 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:36:59.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:59.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:59.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:36:59.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:59.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:59.641 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:59.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:59.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:36:59.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:59.644 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:36:59.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:36:59.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:59.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:59.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:59.665 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:59.662+0000 7fb5fff24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:59.666 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:59.664+0000 7fb5fff24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:59.668 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:36:59.665+0000 7fb5fff24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:00.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:37:00.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:00.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:00.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:00.031 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:37:00.182 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:37:00.179+0000 7fb5fff24ec0 -1 Falling back to public interface 2020-03-11T22:37:00.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:01.112 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:37:01.110+0000 7fb5fff24ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:37:01.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:01.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:01.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:01.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:01.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:01.514 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:37:01.858 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30491,v1:127.0.0.1:6811/30491] [v2:127.0.0.1:6812/30491,v1:127.0.0.1:6813/30491] exists,up d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:37:01.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:01.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:01.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:01.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:01.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:37:02.429 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already in. 2020-03-11T22:37:02.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:02.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:37:03.030 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already in. 2020-03-11T22:37:03.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:03.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:03.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:03.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:03.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:03.041 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:03.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:03.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:03.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:03.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:03.099 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:03.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:03.499 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:37:03.499 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:37:03.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:03.500 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:03.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:03.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:37:03.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:37:03.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:03.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:03.733 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:03.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:03.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:37:03.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:03.741 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:03.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:37:03.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:37:03.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:03.881 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:03.881 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:03.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:03.882 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:03.882 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:03.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:37:03.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:37:03.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:03.883 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:37:04.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:37:04.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:05.244 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:05.245 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:05.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:37:05.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:05.618 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:05.618 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:05.619 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:05.619 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:05.619 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:05.620 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:05.620 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:05.621 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:05.621 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:37:05.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:37:05.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:37:05.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:05.985 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:37:05.985 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:05.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:05.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:37:05.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:05.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:37:05.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:37:05.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:05.987 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:37:06.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:37:06.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:06.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:06.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:06.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:06.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:06.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:06.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:06.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:06.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:06.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:06.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:07.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:07.217 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:07.217 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:07.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:07.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:07.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:07.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:07.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:07.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:07.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:07.791 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:07.791 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:07.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:08.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:08.143 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:08.143 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:08.143 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:08.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:08.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:08.635 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:08.635 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:09.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:09.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:09.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:09.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:09.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:09.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:09.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:09.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:09.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:09.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:09.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:09.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:09.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:09.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:09.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:09.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:09.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:10.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:10.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:10.168 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:10.168 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:10.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:10.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:10.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:10.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:10.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:11.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:11.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:11.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:11.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:11.409 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:11.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:11.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:11.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:11.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:11.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:11.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:11.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:11.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:11.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:11.892 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:11.892 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:12.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:12.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:12.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:12.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:12.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:37:13.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:13.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:13.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:13.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:13.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:13.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:13.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:13.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:13.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:13.527 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:13.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:13.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:13.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:13.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:13.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:13.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:13.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:13.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:13.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:13.999 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:13.999 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:14.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:14.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:14.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:37:14.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:14.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:37:16.085 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:16.085 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:16.086 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:16.086 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:16.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:16.452 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:16.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:16.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:16.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:16.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:16.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:16.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:17.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:17.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:17.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:37:17.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:17.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:37:20.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:20.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:20.608 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:20.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:20.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:20.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:20.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:20.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:20.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:20.971 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:20.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:20.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:21.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:21.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:21.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:21.451 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:21.451 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:21.451 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:21.451 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:21.452 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:21.452 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:21.922 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:21.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:21.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:37:21.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:21.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:37:28.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:28.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:28.325 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:28.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:28.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:28.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:28.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:28.327 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:28.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:28.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:28.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:28.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:29.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:29.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:29.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:29.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:29.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:29.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:37:29.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:29.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:37:36.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:36.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:36.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:36.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:36.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:36.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:36.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:36.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:37.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:37.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:37.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:37.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:37.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:37.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:37.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:37.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:37.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:37.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:37.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:37.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:37.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:38.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:38.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:38.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:37:38.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:37:38.702 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:37:38.702 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "8299e5f9-84ba-4da8-b4a6-89c93ec117e9", 2020-03-11T22:37:38.702 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:37:38.702 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:37:38.702 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:37:38.698854+0000", 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:38.703 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.705 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:37:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:37:38.707 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:37:38.708 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:37:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.710 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:37:38.711 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:35:57.510706Z", 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:35:57.510706Z", 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:37:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.713 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:37:38.715 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:37:38.716 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 39, 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:35:58.272767+0000", 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:37:02.907221+0000", 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:37:01.187777+0000", 2020-03-11T22:37:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:37:02.749364+0000", 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:37:38.718 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 7, 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:37:38.719 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:37:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:36:20.624351+0000", 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 1, 2020-03-11T22:37:38.721 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool", 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 0, 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:37:38.722 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 1, 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 1, 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 1, 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 1, 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:37:38.723 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "29", 2020-03-11T22:37:38.724 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 0, 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 0, 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [], 2020-03-11T22:37:38.725 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:37:38.726 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:37:38.727 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "", 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:37:38.728 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:37:38.729 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "89d0c445-e220-4c12-9579-4ad84b06fa4c", 2020-03-11T22:37:38.730 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 35, 2020-03-11T22:37:38.731 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 38, 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 30, 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:37:38.732 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.733 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.734 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.735 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.736 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.737 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.737 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:37:38.737 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.737 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.737 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:37:38.738 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.739 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.739 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.739 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.739 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:37:38.739 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/30297", 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/30297", 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30297", 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30297", 2020-03-11T22:37:38.740 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:37:38.741 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "86fb20ce-d861-4e6b-99dd-3ca17f77d9c8", 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:37:38.742 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 23, 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 31, 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:38.743 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.744 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.745 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.746 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:38.747 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.748 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.749 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.750 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/25892", 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/25892", 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25892", 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25892", 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.751 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "d5a42e5b-0e5d-4d4e-b380-83acd1693bcc", 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:37:38.752 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 36, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.753 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.754 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.755 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.756 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.757 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.758 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.759 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.760 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/30491", 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/30491", 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30491", 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30491", 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:37:38.761 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:54.829838+0000", 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:05.497276+0000", 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:55.384921+0000", 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:11.901478+0000", 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:37:38.764 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:55.909798+0000", 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:37:38.765 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:18.357978+0000", 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:37:38.766 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:37:38.767 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:37:38.768 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/30297,v1:127.0.0.1:6805/30297]", 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:37:38.769 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:38.770 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:37:38.771 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:37:38.772 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/30297,v1:127.0.0.1:6803/30297]", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30297,v1:127.0.0.1:6809/30297]", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30297,v1:127.0.0.1:6807/30297]", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:37:38.775 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/25892,v1:127.0.0.1:6813/25892]", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:38.776 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:38.777 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:37:38.778 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/25892,v1:127.0.0.1:6811/25892]", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25892,v1:127.0.0.1:6817/25892]", 2020-03-11T22:37:38.779 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25892,v1:127.0.0.1:6815/25892]", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:37:38.780 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.781 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/30491,v1:127.0.0.1:6813/30491]", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:38.783 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:38.784 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:37:38.785 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/30491,v1:127.0.0.1:6811/30491]", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30491,v1:127.0.0.1:6817/30491]", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30491,v1:127.0.0.1:6815/30491]", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:38.786 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:37:38.787 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 39, 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.797 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:37:38.798 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.799 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.800 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:37:38.801 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:37:38.802 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.803 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.804 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.805 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:37:38.806 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:37:38.807 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.808 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.809 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:37:38.810 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.811 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:37:38.812 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:37:38.813 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:37:38.814 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 0, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:37:38.815 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:37:38.816 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:37:38.817 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.818 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:37:38.819 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:37:38.820 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 1, 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:37:38.821 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:38.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:37:38.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:38.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:37:38.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:38.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:37:38.827 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:37:38.828 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 16, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.829 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:38.830 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179200, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33152, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 14, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145713, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393600, 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:37:38.831 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867046400, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33947648, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 403228, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:38.832 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 14706, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221210766 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:38.833 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:37:38.835 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:38.836 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179200, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33152, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 14, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145713, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393600, 2020-03-11T22:37:38.837 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867046400, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33947648, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 403228, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:37:38.838 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:38.839 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:37:38.839 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 14706, 2020-03-11T22:37:38.839 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221210766 2020-03-11T22:37:38.839 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.839 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:37:38.840 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:37:38.841 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:37:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:38.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:37:38.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:38.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:37:38.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:38.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:37:38.848 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19529728, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 143228, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:38.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 16, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:37:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 150323855370 2020-03-11T22:37:38.851 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672975 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 154618822666 2020-03-11T22:37:38.852 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 1 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.853 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.854 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:37:38.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 1, 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:37:38.856 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:35:58.273094+0000", 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:37:38.857 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 301, 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:37:38.858 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:37:38.859 INFO:tasks.workunit.client.0.smithi045.stderr:report 3344957364 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:38.860 INFO:tasks.workunit.client.0.smithi045.stdout:writing non-divergent object existing_4 2020-03-11T22:37:39.055 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:39.056 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 29'100 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:37:39.056 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:37:39.056 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:39.057 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:37:39.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:37:39.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:39.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:39.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:39.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:39.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:39.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:39.175 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:39.176 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:39.176 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:39.176 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:39.176 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:39.176 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:39.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:39.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:39.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:39.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:39.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:39.597 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:39.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855373 2020-03-11T22:37:39.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373' 2020-03-11T22:37:39.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:39.760 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:39.856 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:39.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:39.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373 1-' 2020-03-11T22:37:39.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:39.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:40.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:37:40.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373 1- 2-154618822668' 2020-03-11T22:37:40.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:40.002 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855373 2020-03-11T22:37:40.002 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:40.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:40.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855373 2020-03-11T22:37:40.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:40.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855373 2020-03-11T22:37:40.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2020-03-11T22:37:40.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:40.005 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 150323855373 2020-03-11T22:37:40.368 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855370 -lt 150323855373 2020-03-11T22:37:40.368 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:41.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:41.371 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:41.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855373 -lt 150323855373 2020-03-11T22:37:41.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:41.751 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:41.751 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:41.752 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:41.752 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:41.752 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:41.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:41.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:41.754 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:41.754 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:37:42.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:37:42.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:37:42.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:42.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822668 2020-03-11T22:37:42.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:42.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:42.141 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822668 2020-03-11T22:37:42.141 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:42.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:37:42.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2020-03-11T22:37:42.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:42.142 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 154618822668 2020-03-11T22:37:42.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822668 -lt 154618822668 2020-03-11T22:37:42.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:42.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:42.530 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:43.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:43.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:43.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:43.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:43.358 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:43.358 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:43.358 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:43.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:43.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:43.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:43.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:43.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:43.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:43.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:43.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:43.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:43.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:43.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:43.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:43.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:43.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:44.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:44.280 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:44.280 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:44.281 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:44.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:44.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:44.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:44.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:45.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:45.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:45.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:45.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:45.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:45.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:45.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:45.425 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:45.425 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:45.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:45.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:45.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:45.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:45.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:45.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:45.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:45.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:46.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:46.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:46.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:46.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:46.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:46.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:46.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:46.248 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:46.248 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:46.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:46.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:46.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:46.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:46.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:47.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:47.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:47.134 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:47.135 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:47.135 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:47.135 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:47.135 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:47.135 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:47.487 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:47.487 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:47.488 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:47.488 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:48.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:48.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:48.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:48.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:48.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:37:49.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:49.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:49.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:49.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:49.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:49.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:49.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:49.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:49.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:49.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:49.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:49.602 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:50.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:50.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:50.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:50.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:50.556 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:50.556 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:50.556 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:37:50.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:50.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:37:52.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:52.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:52.158 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:52.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:52.519 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:52.519 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:52.519 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:53.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:53.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:53.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:53.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:53.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:53.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:53.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:37:53.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:53.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:37:56.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:56.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:56.676 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:56.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:56.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:56.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:56.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:56.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:57.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:57.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:57.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:57.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:57.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:57.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:57.515 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:57.515 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:57.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:57.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:57.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:37:57.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:57.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:38:04.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:04.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:04.387 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:04.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:04.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:04.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:04.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:04.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:04.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:04.751 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:04.752 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:04.752 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:05.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:05.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:05.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:05.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:05.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:05.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:05.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:38:05.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:05.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:38:13.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:13.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:13.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:13.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:13.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:13.386 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:13.387 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:13.387 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:13.861 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:14.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:14.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:14.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:38:14.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:38:14.797 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:38:14.797 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "8299e5f9-84ba-4da8-b4a6-89c93ec117e9", 2020-03-11T22:38:14.797 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:38:14.797 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:38:14.793870+0000", 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:14.798 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.799 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:38:14.800 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:38:14.801 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:14.802 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:38:14.802 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:38:14.802 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 2 2020-03-11T22:38:14.802 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.803 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:38:14.804 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.805 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:35:57.510706Z", 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:35:57.510706Z", 2020-03-11T22:38:14.807 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:38:14.808 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.809 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:38:14.810 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:38:14.811 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.812 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 39, 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:35:58.272767+0000", 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:37:02.907221+0000", 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:37:01.187777+0000", 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:37:02.749364+0000", 2020-03-11T22:38:14.813 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:38:14.814 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 7, 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:38:14.815 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:38:14.816 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:36:20.624351+0000", 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 1, 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool", 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:38:14.817 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 0, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 1, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:38:14.818 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 1, 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 1, 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 1, 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:38:14.819 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "29", 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:38:14.820 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 0, 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 0, 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [], 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:38:14.821 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:38:14.822 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:38:14.823 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "", 2020-03-11T22:38:14.824 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:38:14.824 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:38:14.824 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.824 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:38:14.824 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:38:14.825 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:38:14.825 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:38:14.825 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:38:14.825 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:38:14.825 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 0, 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:38:14.826 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "89d0c445-e220-4c12-9579-4ad84b06fa4c", 2020-03-11T22:38:14.827 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 35, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 38, 2020-03-11T22:38:14.828 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 30, 2020-03-11T22:38:14.829 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:38:14.829 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:38:14.829 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.829 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.829 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.830 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.831 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:38:14.832 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30297 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/30297", 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/30297", 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30297", 2020-03-11T22:38:14.837 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30297", 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "86fb20ce-d861-4e6b-99dd-3ca17f77d9c8", 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:38:14.839 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 23, 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 31, 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.840 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.842 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:14.843 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.844 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:14.845 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 25892 2020-03-11T22:38:14.847 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/25892", 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/25892", 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25892", 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25892", 2020-03-11T22:38:14.848 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "d5a42e5b-0e5d-4d4e-b380-83acd1693bcc", 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:38:14.849 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 36, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:38:14.850 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:14.851 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.852 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.853 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.854 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.856 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.857 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 30491 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.858 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/30491", 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/30491", 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30491", 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30491", 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:38:14.859 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:38:14.860 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:54.829838+0000", 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:05.497276+0000", 2020-03-11T22:38:14.861 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:55.384921+0000", 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:38:14.862 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:11.901478+0000", 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:38:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:36:55.909798+0000", 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:18.357978+0000", 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.865 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:38:14.866 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.867 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.868 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/30297,v1:127.0.0.1:6805/30297]", 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:38:14.869 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:14.870 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:38:14.871 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:14.872 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:14.872 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:14.872 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:38:14.873 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:38:14.873 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:14.873 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:38:14.873 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:38:14.873 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/30297,v1:127.0.0.1:6803/30297]", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30297,v1:127.0.0.1:6809/30297]", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30297,v1:127.0.0.1:6807/30297]", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:38:14.874 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:14.875 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:38:14.876 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/25892,v1:127.0.0.1:6813/25892]", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:14.877 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:14.878 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:38:14.879 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/25892,v1:127.0.0.1:6811/25892]", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:38:14.880 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25892,v1:127.0.0.1:6817/25892]", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25892,v1:127.0.0.1:6815/25892]", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:38:14.882 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/30491,v1:127.0.0.1:6813/30491]", 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:38:14.883 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:14.884 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:38:14.885 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:38:14.886 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/30491,v1:127.0.0.1:6811/30491]", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30491,v1:127.0.0.1:6817/30491]", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30491,v1:127.0.0.1:6815/30491]", 2020-03-11T22:38:14.887 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:38:14.888 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:38:14.888 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:14.888 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:14.888 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:14.888 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:38:14.889 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 39, 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:38:14.891 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.892 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:38:14.893 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:38:14.894 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.895 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.896 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:38:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:38:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.899 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:38:14.900 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.901 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.902 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:38:14.903 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:38:14.904 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.905 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.907 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:38:14.908 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.909 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:38:14.910 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:38:14.911 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:38:14.911 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.911 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:38:14.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:38:14.916 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:38:14.917 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:38:14.918 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 0, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:38:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:38:14.920 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:38:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 1, 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 47947, 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:38:14.925 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:14.927 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:14.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:14.929 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:14.930 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:38:14.931 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:38:14.932 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 17, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:38:14.933 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:38:14.934 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179200, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33152, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 14, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145713, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393600, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:38:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867046400, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33947648, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 402522, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:38:14.936 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 14706, 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221210766 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:14.937 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:14.938 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.939 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:14.940 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:38:14.941 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179200, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33152, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 14, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145713, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311393600, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318867046400, 2020-03-11T22:38:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 33947648, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 402522, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 14706, 2020-03-11T22:38:14.943 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221210766 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:38:14.944 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:14.945 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:38:14.946 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 47947, 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:38:14.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:38:14.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:14.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:38:14.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:14.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:14.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:14.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:14.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:14.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:38:14.953 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19529728, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 142522, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:38:14.954 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 17, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 2, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 2, 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:38:14.955 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 150323855380 2020-03-11T22:38:14.956 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672975 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:38:14.957 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 154618822675 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 1 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.960 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 1, 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.961 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:38:14.962 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:35:58.273094+0000", 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 380, 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:38:14.963 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:38:14.964 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:38:14.964 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:38:14.965 INFO:tasks.workunit.client.0.smithi045.stderr:report 2285665163 2020-03-11T22:38:14.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:38:14.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:38:14.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:14.965 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery 2020-03-11T22:38:15.149 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:15.149 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:15.149 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:15.149 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:15.151 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:15.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:15.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:38:15.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:15.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:15.160 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:15.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:38:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:38:15.256 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:15.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:15.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:15.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:38:15.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:15.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:38:15.362 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:15.368 INFO:tasks.workunit.client.0.smithi045.stdout:revive divergent 1 2020-03-11T22:38:15.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:38:15.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:15.710 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:15.710 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:15.711 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:15.712 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:15.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:38:16.312 INFO:tasks.workunit.client.0.smithi045.stderr:noup is set 2020-03-11T22:38:16.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:16.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:16.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:16.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:16.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:16.325 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:16.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:16.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:16.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:16.328 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:16.328 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:38:16.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:16.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:16.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:16.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:16.351 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:16.348+0000 7fb2ffe78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:16.353 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:16.351+0000 7fb2ffe78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:16.355 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:16.353+0000 7fb2ffe78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:16.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:38:16.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:38:16.871 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:16.868+0000 7fb2ffe78ec0 -1 Falling back to public interface 2020-03-11T22:38:17.747 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:17.745+0000 7fb2ffe78ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:38:21.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:38:21.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:21.691 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery divergent 2020-03-11T22:38:22.044 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:22.045 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:22.045 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:22.045 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:22.046 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:22.054 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:38:22.054 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:22.055 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:22.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:38:22.152 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:22.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:38:23.001 INFO:tasks.workunit.client.0.smithi045.stderr:noup is unset 2020-03-11T22:38:23.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:38:23.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:23.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:23.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:23.012 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:38:23.380 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/30297,v1:127.0.0.1:6803/30297] [v2:127.0.0.1:6804/30297,v1:127.0.0.1:6805/30297] exists,up 89d0c445-e220-4c12-9579-4ad84b06fa4c 2020-03-11T22:38:23.380 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:38:23.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:23.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:23.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:23.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:23.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:23.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:23.760 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33038,v1:127.0.0.1:6819/33038] [v2:127.0.0.1:6820/33038,v1:127.0.0.1:6821/33038] exists,up 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:38:23.760 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:38:23.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:23.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:23.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:23.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:23.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:23.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30491,v1:127.0.0.1:6811/30491] [v2:127.0.0.1:6812/30491,v1:127.0.0.1:6813/30491] exists,up d5a42e5b-0e5d-4d4e-b380-83acd1693bcc 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:24.498 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.525 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:24.525 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:24.525 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.527 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:24.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:38:24.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:24.527 INFO:tasks.workunit.client.0.smithi045.stdout:wait for peering 2020-03-11T22:38:24.904 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.904 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:03.121189+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:24.905 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:24.905 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.906 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:24.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:38:31.140 INFO:tasks.workunit.client.0.smithi045.stdout:killing divergent 1 2020-03-11T22:38:31.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:38:31.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:31.495 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:31.495 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:23.298114+0000 39'101 45:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:31.495 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:31.495 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:31.497 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:31.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:38:31.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:31.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:31.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:31.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:31.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:31.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:31.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17438.out 2020-03-11T22:38:31.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17438.out 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:31.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17438.out 2020-03-11T22:38:32.235 INFO:tasks.workunit.client.0.smithi045.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (29'90,45'102] local-lis/les=43/45 n=101 ec=23/23 lis/c=43/23 les/c/f=45/24/0 sis=43 pruub=7.826878457s) 2020-03-11T22:38:32.254 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:32.274 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:32.288 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:32.291 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:32.291 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:38:32.297 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:38:32.298 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:32.304 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:38:32.306 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:38:32.307 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:38:32.307 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:38:32.310 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:32.314 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:38:32.322 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:38:32.322 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:38:32.330 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:38:32.331 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:38:32.332 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:38:32.335 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:38:32.336 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:38:32.336 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:32.340 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:32.345 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:38:32.354 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:32.355 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:32.356 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:32.356 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:38:32.356 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:32.357 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:38:32.358 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:32.358 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:32.359 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:38:32.365 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:38:32.365 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:38:32.366 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:32.366 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:38:32.372 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:38:32.378 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:38:32.379 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:38:32.379 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:32.380 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:32.380 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:32.380 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:38:32.381 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:38:32.381 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:38:32.381 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:32.382 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:38:32.382 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:32.385 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:32.387 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:32.387 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:32.387 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:38:32.388 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:38:32.388 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:38:32.389 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:32.389 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:32.389 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:38:32.395 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:32.396 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:32.396 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:38:32.397 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:38:32.397 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:32.397 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:32.399 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:38:32.399 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:32.400 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:32.400 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:38:32.400 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:38:32.401 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:32.402 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:38:32.402 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:32.402 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:38:32.402 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:38:32.403 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:38:32.403 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:32.403 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:32.404 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:32.404 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:32.404 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:32.406 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:32.406 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:38:32.409 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:38:32.410 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:32.411 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:32.411 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:32.411 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:38:32.411 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:32.412 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stderr:Export successful 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stdout: marking collection for removal 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stdout:setting '_remove' omap key 2020-03-11T22:38:32.413 INFO:tasks.workunit.client.0.smithi045.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:38:32.481 INFO:tasks.workunit.client.0.smithi045.stderr:Remove successful 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17438.out 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:32.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17438.out 2020-03-11T22:38:33.396 INFO:tasks.workunit.client.0.smithi045.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={23=1}} deleted_pools ) 2020-03-11T22:38:33.396 INFO:tasks.workunit.client.0.smithi045.stdout:Importing pgid 1.0 2020-03-11T22:38:33.396 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:33.396 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:33.447 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.455 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:33.455 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.463 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:33.463 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.471 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:38:33.471 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.480 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:38:33.480 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.488 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:33.488 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.496 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:38:33.497 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.505 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:38:33.505 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.513 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:38:33.513 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.521 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:38:33.522 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.530 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:33.530 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.538 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:38:33.538 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.546 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:38:33.547 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.736 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:38:33.737 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.744 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:38:33.744 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.752 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:38:33.752 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.760 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:38:33.761 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.769 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:38:33.769 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.777 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:38:33.777 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.786 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:33.786 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.794 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:33.794 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.802 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:38:33.802 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.811 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:33.811 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.819 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:33.819 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.827 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:33.828 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.836 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:38:33.836 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.867 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:33.867 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.953 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:38:33.954 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.963 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:33.964 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.972 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:33.972 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.980 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:38:33.980 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.988 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:38:33.988 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:33.996 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:38:33.997 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.005 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:34.005 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.013 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:38:34.014 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.022 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:38:34.022 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.030 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:38:34.030 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.039 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:38:34.039 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.047 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:34.047 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.112 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:34.112 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.122 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:38:34.122 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.133 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:38:34.133 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.138 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:38:34.139 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.147 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:34.147 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.155 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:38:34.155 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.164 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:34.164 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.172 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:34.172 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.180 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:34.180 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.188 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:34.189 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.197 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:38:34.197 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.205 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:38:34.205 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.214 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:38:34.214 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.222 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:34.222 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.253 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:34.253 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.264 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:38:34.264 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.272 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:34.272 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.282 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:34.282 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.289 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:38:34.289 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.297 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:38:34.298 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.306 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:34.306 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.314 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:34.314 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.322 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:38:34.323 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.364 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:34.364 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.399 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:34.399 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.431 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:38:34.431 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.439 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:38:34.439 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.462 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:34.462 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.473 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:38:34.473 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.481 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:34.481 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.489 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:38:34.490 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.498 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:38:34.498 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.506 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:38:34.506 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.514 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:34.515 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.523 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:34.523 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.531 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:34.531 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.539 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:34.540 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.548 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:34.548 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.556 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:34.556 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.564 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:34.565 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.606 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:34.607 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.679 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:38:34.679 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.689 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:34.690 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.698 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:34.698 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.706 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:34.707 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.715 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:34.715 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.723 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:38:34.723 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.731 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:38:34.732 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.740 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:34.740 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.748 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:34.748 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.756 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:34.757 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.765 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:38:34.765 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.773 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:34.773 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.782 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:38:34.782 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.790 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:38:34.790 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.850 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:34.851 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:38:34.865 INFO:tasks.workunit.client.0.smithi045.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (29'90,45'102] local-lis/les=43/45 n=101 ec=23/23 lis/c=43/23 les/c/f=45/24/0 sis=43 pruub=7.826878457s) 2020-03-11T22:38:34.873 INFO:tasks.workunit.client.0.smithi045.stdout:Import successful 2020-03-11T22:38:35.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:38:35.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:35.396 INFO:tasks.workunit.client.0.smithi045.stdout:reviving divergent 1 2020-03-11T22:38:35.749 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:35.749 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:23.298114+0000 39'101 45:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:35.750 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:35.750 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:35.751 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:35.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:35.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:35.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:35.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:35.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:38:35.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:35.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:35.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:35.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:35.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:35.767 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:35.770 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:38:35.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:35.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:35.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:35.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:35.789 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:35.786+0000 7f87357c3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:35.791 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:35.788+0000 7f87357c3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:35.793 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:35.790+0000 7f87357c3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:36.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:38:36.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:36.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:36.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:36.130 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:38:36.310 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:36.307+0000 7f87357c3ec0 -1 Falling back to public interface 2020-03-11T22:38:36.519 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33038,v1:127.0.0.1:6819/33038] [v2:127.0.0.1:6820/33038,v1:127.0.0.1:6821/33038] exists,up 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:38:36.519 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:38:36.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:36.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:36.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:36.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:36.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:36.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:36.907 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33038,v1:127.0.0.1:6819/33038] [v2:127.0.0.1:6820/33038,v1:127.0.0.1:6821/33038] exists,up 86fb20ce-d861-4e6b-99dd-3ca17f77d9c8 2020-03-11T22:38:36.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:36.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:36.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:36.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:38:37.146 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:38:37.142+0000 7f87357c3ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:38:56.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:38:56.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:56.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:56.918 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:56.918 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:56.918 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:56.918 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:38:56.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:56.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:38:57.019 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:38:57.019 INFO:tasks.workunit.client.0.smithi045.stdout: "ops": [], 2020-03-11T22:38:57.019 INFO:tasks.workunit.client.0.smithi045.stdout: "num_ops": 0 2020-03-11T22:38:57.019 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:38:57.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:38:57.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:57.028 INFO:tasks.workunit.client.0.smithi045.stdout:allowing recovery 2020-03-11T22:38:57.363 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:57.363 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:39.000205+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:57.363 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:57.363 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:57.364 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:57.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:57.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:38:57.500 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:57.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:57.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:38:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:57.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:57.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:38:57.772 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:57.781 INFO:tasks.workunit.client.0.smithi045.stdout:reading divergent objects 2020-03-11T22:38:57.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:38:57.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:58.122 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:58.122 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:39.000205+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:20.674524+0000 0'0 2020-03-11T22:36:20.674524+0000 0 2020-03-11T22:38:58.123 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:38:58.123 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:58.124 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:38:58.133 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:38:58.134 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:38:58.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:58.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:38:58.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:58.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:38:58.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:58.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:38:58.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:58.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:38:58.252 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:38:58.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:58.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi045.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:38:58.281 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:58.281 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:38:58.309 INFO:tasks.workunit.client.0.smithi045.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:38:58.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:38:58.312 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:58.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:58.315 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-03-11T22:36:44.073163+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:58.315 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-03-11T22:36:44.073163+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'105 last_divergent_update: 29'105 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:37:39.090991+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4527.0:1 2020-03-11T22:36:44.051879+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4527.0:1 2020-03-11T22:36:44.051879+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.316 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:44.073022+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:44.073022+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:38:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:44.073022+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2020-03-11T22:38:58.318 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2020-03-11T22:36:44.059402+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2020-03-11T22:36:44.059402+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2020-03-11T22:38:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2020-03-11T22:36:44.059402+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:38:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2020-03-11T22:38:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:36:44.069112+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:36:44.069112+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:58.321 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:38:58.321 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:38:58.321 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:58.321 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:36:44.069112+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:36:44.065753+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:36:44.065753+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:23.131+0000 7fb2de033700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:58.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:58.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:58.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:38:58.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17438.out 2020-03-11T22:38:58.324 INFO:tasks.workunit.client.0.smithi045.stdout:success 2020-03-11T22:38:58.325 INFO:tasks.workunit.client.0.smithi045.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17438.out': No such file or directory 2020-03-11T22:38:58.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:38:58.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:38:58.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:38:58.835 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' does not exist 2020-03-11T22:38:58.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:38:58.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:58.844 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:58.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:58.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:58.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:58.951 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:58.952 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:58.953 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:58.953 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:58.953 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:58.956 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:58.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:38:58.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:38:58.958 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:38:58.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:38:58.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:38:58.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:38:58.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:58.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:38:58.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:38:58.962 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:58.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:38:58.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:38:58.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:38:58.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:38:58.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:58.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:58.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:38:58.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:38:58.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:38:58.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:38:58.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:38:58.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:58.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:58.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:38:58.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:38:58.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:38:58.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:58.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:38:58.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:38:58.984 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:58.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:38:58.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:38:58.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:38:58.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:38:58.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:58.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:58.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:38:58.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:38:58.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:38:58.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:38:58.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:38:58.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:58.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:58.988 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17438 2020-03-11T22:38:58.990 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:38:58.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:38:58.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:38:58.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:38:58.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:38:58.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:38:58.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:38:58.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:38:58.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:38:58.991 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:38:58.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:38:58.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:38:58.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:38:58.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:38:58.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:38:58.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:38:59.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:38:59.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:59.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:59.277 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:59.277 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:59.277 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:59.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:38:59.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:38:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:38:59.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:38:59.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:38:59.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:38:59.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:38:59.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:38:59.447 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:38:59.447 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:38:59.447 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:38:59.447 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:59.448 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:59.448 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:59.448 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:38:59.448 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:38:59.448 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get fsid 2020-03-11T22:38:59.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:38:59.541 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:38:59.542 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:38:59.542 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:59.543 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:59.543 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:38:59.543 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:38:59.543 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:38:59.543 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get mon_host 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:38:59.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:38:59.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:39:00.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:39:00.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:00.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:00.564 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:00.565 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:00.565 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:00.565 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:00.565 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:00.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:00.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:00.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:39:00.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:00.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:00.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:39:00.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:00.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:00.600 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:00.600 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:00.600 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:00.600 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:00.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:00.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:00.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:00.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:00.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:00.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:00.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:00.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:00.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:00.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:00.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:00.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:00.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:00.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:39:00.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 7cc010f5-1143-4f6a-94c4-303fedff664a' 2020-03-11T22:39:00.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:00.604 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:39:00.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCEaGle+8eQJBAAZ4L8nFqXEruF2PWh6fiisw== 2020-03-11T22:39:00.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCEaGle+8eQJBAAZ4L8nFqXEruF2PWh6fiisw=="}' 2020-03-11T22:39:00.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7cc010f5-1143-4f6a-94c4-303fedff664a -i td/divergent-priors/0/new.json 2020-03-11T22:39:00.969 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:39:00.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:39:00.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCEaGle+8eQJBAAZ4L8nFqXEruF2PWh6fiisw== --osd-uuid 7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:39:01.010 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.007+0000 7f00edfcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:01.011 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.009+0000 7f00edfcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:01.013 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.010+0000 7f00edfcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:01.013 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.011+0000 7f00edfcbec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:01.013 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.011+0000 7f00edfcbec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:01.014 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.011+0000 7f00edfcbec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:01.051 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:01.049+0000 7f00edfcbec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:39:04.146 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:39:04.146 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:04.147 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:39:04.147 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:39:04.147 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:04.649 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:39:04.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:39:04.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:04.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:04.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:04.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:04.677 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:04.675+0000 7f7490217ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:04.711 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:04.709+0000 7f7490217ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:04.713 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:04.710+0000 7f7490217ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:05.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:05.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:05.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:05.014 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:39:05.228 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:05.225+0000 7f7490217ec0 -1 Falling back to public interface 2020-03-11T22:39:05.426 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:05.994 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:05.991+0000 7f7490217ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:39:06.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:06.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:06.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:06.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:06.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:06.428 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:39:06.852 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34534,v1:127.0.0.1:6803/34534] [v2:127.0.0.1:6804/34534,v1:127.0.0.1:6805/34534] exists,up 7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:06.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:06.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:06.855 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:06.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:06.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:06.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:06.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:06.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:06.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:39:06.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 f9d3fe39-2937-4660-9a35-ada2db8fe0c4' 2020-03-11T22:39:06.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:06.859 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:39:06.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCKaGleCc/VMxAAfGjMIUQvTtSxkbxe3T4pBQ== 2020-03-11T22:39:06.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCKaGleCc/VMxAAfGjMIUQvTtSxkbxe3T4pBQ=="}' 2020-03-11T22:39:06.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f9d3fe39-2937-4660-9a35-ada2db8fe0c4 -i td/divergent-priors/1/new.json 2020-03-11T22:39:07.353 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:39:07.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:39:07.363 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCKaGleCc/VMxAAfGjMIUQvTtSxkbxe3T4pBQ== --osd-uuid f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:39:07.394 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.391+0000 7f4151d99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:07.408 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.405+0000 7f4151d99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:07.409 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.407+0000 7f4151d99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:07.410 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.407+0000 7f4151d99ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:07.410 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.407+0000 7f4151d99ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:07.410 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.407+0000 7f4151d99ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:07.484 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:07.480+0000 7f4151d99ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:39:10.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:39:10.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:10.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:39:10.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:10.589 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:39:11.075 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:39:11.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:39:11.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:11.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:11.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:11.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:11.103 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:11.101+0000 7f5c12668ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.105 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:11.103+0000 7f5c12668ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.107 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:11.104+0000 7f5c12668ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:39:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:11.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:11.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:11.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:11.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:11.450 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:39:11.625 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:11.622+0000 7f5c12668ec0 -1 Falling back to public interface 2020-03-11T22:39:11.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:12.326 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:12.323+0000 7f5c12668ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:39:12.814 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:12.814 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:12.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:12.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:12.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:12.815 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:39:13.215 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34889,v1:127.0.0.1:6811/34889] [v2:127.0.0.1:6812/34889,v1:127.0.0.1:6813/34889] exists,up f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:39:13.215 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:39:13.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:13.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:13.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:39:13.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:13.218 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:13.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:13.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:39:13.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:13.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:39:13.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 ad4265b4-f9e4-4561-9f43-b939c3dfabe7' 2020-03-11T22:39:13.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:13.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCRaGleEj3KDRAAY66FvX/Kid7Ryp6VcvvSuQ== 2020-03-11T22:39:13.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCRaGleEj3KDRAAY66FvX/Kid7Ryp6VcvvSuQ=="}' 2020-03-11T22:39:13.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ad4265b4-f9e4-4561-9f43-b939c3dfabe7 -i td/divergent-priors/2/new.json 2020-03-11T22:39:13.710 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:39:13.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:39:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCRaGleEj3KDRAAY66FvX/Kid7Ryp6VcvvSuQ== --osd-uuid ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:39:13.745 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.742+0000 7fb7d85d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.747 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.744+0000 7fb7d85d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.748 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.746+0000 7fb7d85d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.749 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.746+0000 7fb7d85d9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:13.749 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.746+0000 7fb7d85d9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:13.749 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.746+0000 7fb7d85d9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:13.795+0000 7fb7d85d9ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:39:17.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:39:17.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:17.020 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:39:17.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:39:17.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:17.510 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:39:17.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:39:17.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:17.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:17.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:17.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:17.538 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:17.535+0000 7f46847a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:17.540 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:17.537+0000 7f46847a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:17.541 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:17.539+0000 7f46847a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:17.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:39:17.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:17.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:17.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:17.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:17.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:17.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:17.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:17.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:17.874 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:39:18.056 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:18.054+0000 7f46847a4ec0 -1 Falling back to public interface 2020-03-11T22:39:18.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:18.823 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:18.820+0000 7f46847a4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:39:19.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:19.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:19.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:19.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:19.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:19.296 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:39:19.681 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35244,v1:127.0.0.1:6819/35244] [v2:127.0.0.1:6820/35244,v1:127.0.0.1:6821/35244] exists,up ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:39:19.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:19.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:19.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:19.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:39:20.383 INFO:tasks.workunit.client.0.smithi045.stderr:noout is set 2020-03-11T22:39:20.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:39:21.068 INFO:tasks.workunit.client.0.smithi045.stderr:noin is set 2020-03-11T22:39:21.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:39:21.802 INFO:tasks.workunit.client.0.smithi045.stderr:nodown is set 2020-03-11T22:39:21.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:39:21.811 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:39:22.340 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:39:22.348 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:39:23.349 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:39:23.998 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 3 2020-03-11T22:39:24.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:39:24.833 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 min_size to 2 2020-03-11T22:39:24.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:39:25.460 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:39:25.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:39:25.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:25.468 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:25.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:25.835 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:39:25.835 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:39:25.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:25.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:25.836 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:25.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:39:25.972 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:39:25.972 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:25.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:26.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:39:26.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:39:26.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:26.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:26.244 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:39:26.244 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:39:26.244 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:26.244 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:26.245 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:26.245 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:26.245 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:26.246 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:26.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:39:26.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:39:26.247 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:26.247 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:39:26.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:39:26.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:26.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:26.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:26.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:26.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:26.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:26.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:39:26.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:39:26.612 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:26.612 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:39:26.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672963 -lt 42949672965 2020-03-11T22:39:26.988 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:28.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:28.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:28.368 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:39:28.368 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:28.368 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:28.368 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:28.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:28.369 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:28.369 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:28.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:39:28.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:39:28.370 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:28.371 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:39:28.738 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:28.739 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:39:28.740 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:28.740 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:28.740 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:28.740 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:28.741 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:28.836 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:29.197 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:29.197 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:39:29.198 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:39:29.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:29.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:29.198 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:29.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:39:29.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:39:29.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:29.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:29.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:39:29.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:39:29.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:29.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:29.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:39:29.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:39:29.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:29.607 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:29.607 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:29.608 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:29.608 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:29.608 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:29.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:39:29.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:39:29.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:29.610 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:39:30.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:39:30.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:31.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:31.006 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:31.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:39:31.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:31.385 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:39:31.385 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:31.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:31.386 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:39:31.386 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:31.387 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:39:31.387 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:39:31.388 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:31.388 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:39:31.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:39:31.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:31.758 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:31.758 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:31.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:31.759 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:31.759 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:31.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:39:31.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:39:31.760 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:31.761 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:39:32.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:39:32.156 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:32.157 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:32.157 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:32.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:32.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:32.625 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:32.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:32.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:32.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:32.977 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:33.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:33.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:33.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:33.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:39:33.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:39:33.801 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:39:33.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:39:33.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:39:33.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:33.810 INFO:tasks.workunit.client.0.smithi045.stdout:primary and soon to be divergent is 1 2020-03-11T22:39:34.153 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:34.154 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:39:23.182125+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:39:34.154 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:39:34.154 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:34.155 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:39:34.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:39:34.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:34.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:39:34.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:39:34.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:34.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:39:34.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:39:34.167 INFO:tasks.workunit.client.0.smithi045.stdout:writing initial objects 2020-03-11T22:39:34.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:39:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:39:34.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:39:34.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:39:34.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:39:34.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.558 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:39:34.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:39:34.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:39:34.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:39:34.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:39:34.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:39:34.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:39:34.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:39:34.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:34.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:39:35.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:39:35.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:39:35.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:39:35.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:39:35.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:39:35.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:39:35.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:39:35.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:39:35.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:39:35.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:39:35.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:39:35.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:39:35.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:39:35.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:39:35.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:35.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:39:36.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:39:36.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:39:36.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:39:36.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.307 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:39:36.347 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.347 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:39:36.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:39:36.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:39:36.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:39:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:39:36.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:39:36.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:39:36.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.791 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:39:36.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:39:36.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:36.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:39:37.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:39:37.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:39:37.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:39:37.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:39:37.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:39:37.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:39:37.378 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:39:37.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:39:37.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:39:37.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:39:37.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:39:37.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:39:37.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:39:37.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:39:37.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:39:37.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:39:37.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:37.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:39:38.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:39:38.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:39:38.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:39:38.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:39:38.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:39:38.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:39:38.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:39:38.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:39:38.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:39:38.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:39:38.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:39:38.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:39:38.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:39:38.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:39:38.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:39:38.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:39:38.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:39:38.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.952 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:39:38.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:38.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:39:39.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:39:39.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:39:39.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:39:39.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:39:39.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:39:39.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:39:39.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:39:39.602 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:39:39.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:39:39.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:39:39.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:39:39.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:39:39.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:39.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:39:40.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:39:40.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:39:40.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:39:40.309 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.309 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:39:40.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:39:40.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:40.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:39:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:40.486 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:40.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:40.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:40.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:40.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:40.545 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:40.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:40.910 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:39:40.911 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:39:40.911 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:40.911 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:40.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:41.059 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:39:41.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:39:41.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:41.060 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:41.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:39:41.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:39:41.197 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:41.197 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:41.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:39:41.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:39:41.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:41.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:41.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:41.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:41.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:41.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:41.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:39:41.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:39:41.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:41.335 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:39:41.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:39:41.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:42.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:42.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:43.077 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:39:43.077 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:43.078 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:43.078 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:43.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:43.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:43.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:43.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:39:43.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:39:43.080 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:39:43.081 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:43.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:39:43.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:43.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:39:43.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:43.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:43.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:39:43.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:43.447 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:39:43.447 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:39:43.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:43.447 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:39:43.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:39:43.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:43.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:43.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:44.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:44.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:44.318 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:44.318 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:44.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:44.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:44.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:44.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:44.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:44.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:44.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:44.689 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:45.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:45.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:45.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:45.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:39:45.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:39:45.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:39:45.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:39:45.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:39:45.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:39:45.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:39:45.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:45.555 INFO:tasks.workunit.client.0.smithi045.stdout:blackholing osds 0 2 2020-03-11T22:39:45.899 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:45.899 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:39:23.182125+0000 31'100 31:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:39:45.900 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:39:45.900 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:45.901 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:39:45.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:45.910 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:39:45.910 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:45.911 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:39:46.002 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:39:46.002 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:46.002 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:39:46.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:46.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:39:46.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:46.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:39:46.102 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:39:46.102 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:46.102 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:39:46.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:39:46.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:39:46.110 INFO:tasks.workunit.client.0.smithi045.stdout:create new divergent objects 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:39:46.111 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:39:46.112 INFO:tasks.workunit.client.0.smithi045.stdout:writing divergent objects 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:39:46.113 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:39:46.114 INFO:tasks.workunit.client.0.smithi045.stdout:remove divergent objects 2020-03-11T22:39:46.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:46.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:39:46.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:39:46.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:46.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:39:46.116 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:39:46.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:39:46.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:39:46.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:39:56.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:39:56.122 INFO:tasks.workunit.client.0.smithi045.stderr:rados: no process found 2020-03-11T22:39:56.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:39:56.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:56.122 INFO:tasks.workunit.client.0.smithi045.stdout:killing all the osds 2020-03-11T22:39:56.472 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:56.472 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:39:23.182125+0000 31'106 31:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:39:56.472 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:39:56.473 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:56.474 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:39:56.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:39:56.483 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:56.483 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:56.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:56.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:56.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:56.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:56.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:56.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:39:57.096 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already down. 2020-03-11T22:39:57.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:57.105 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:39:57.615 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already down. 2020-03-11T22:39:57.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:57.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:39:58.132 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already down. 2020-03-11T22:39:58.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:58.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:39:58.675 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already out. 2020-03-11T22:39:58.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:58.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:39:59.241 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already out. 2020-03-11T22:39:59.248 INFO:tasks.workunit.client.0.smithi045.stdout:bringing up non_divergent 0 2 2020-03-11T22:39:59.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:39:59.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:59.586 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:59.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:39:23.182125+0000 31'106 31:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:39:59.586 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:39:59.586 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:59.587 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:39:59.593 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:59.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:59.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:59.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:59.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:59.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:59.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:59.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:59.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:59.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:59.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:59.609 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:39:59.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:59.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:59.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:59.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:59.613 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:39:59.622 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:59.620+0000 7f31160d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:59.624 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:59.622+0000 7f31160d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:59.626 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:39:59.624+0000 7f31160d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:59.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:59.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:59.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:59.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:59.963 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:40:00.146 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:00.143+0000 7f31160d3ec0 -1 Falling back to public interface 2020-03-11T22:40:00.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:01.175 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:01.172+0000 7f31160d3ec0 -1 osd.0 31 log_to_monitors {default=true} 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:01.336 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:40:01.715 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/39352,v1:127.0.0.1:6803/39352] [v2:127.0.0.1:6804/39352,v1:127.0.0.1:6805/39352] exists,up 7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:40:01.715 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:40:01.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:01.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:01.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:01.718 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:40:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:01.721 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:40:01.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:40:01.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:01.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:01.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:01.742 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:01.740+0000 7f05be182ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:01.743 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:01.741+0000 7f05be182ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:01.745 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:01.743+0000 7f05be182ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:02.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:40:02.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:02.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:02.115 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:40:02.266 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:02.263+0000 7f05be182ec0 -1 Falling back to public interface 2020-03-11T22:40:02.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:03.148 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:40:03.145+0000 7f05be182ec0 -1 osd.2 31 log_to_monitors {default=true} 2020-03-11T22:40:03.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:03.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:03.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:03.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:03.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:03.519 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:40:03.895 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6810/39546,v1:127.0.0.1:6811/39546] [v2:127.0.0.1:6812/39546,v1:127.0.0.1:6813/39546] exists,up ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:40:03.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:03.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:03.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:03.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:03.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:40:04.526 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already in. 2020-03-11T22:40:04.534 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:04.534 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:40:05.086 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already in. 2020-03-11T22:40:05.094 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:05.095 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:05.096 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:05.096 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:05.096 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:05.096 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:05.097 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:05.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:05.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:05.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:05.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:05.155 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:05.549 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:05.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789955 2020-03-11T22:40:05.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955' 2020-03-11T22:40:05.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:05.686 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:05.781 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:05.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:05.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955 1-' 2020-03-11T22:40:05.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:05.791 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:05.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757251 2020-03-11T22:40:05.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955 1- 2-163208757251' 2020-03-11T22:40:05.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:05.926 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-158913789955 2020-03-11T22:40:05.926 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:05.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:05.927 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-158913789955 2020-03-11T22:40:05.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:05.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789955 2020-03-11T22:40:05.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 158913789955' 2020-03-11T22:40:05.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:05.929 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 158913789955 2020-03-11T22:40:06.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789955 -lt 158913789955 2020-03-11T22:40:06.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:06.310 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:06.311 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:06.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:06.312 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:06.312 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:06.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:06.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:06.313 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:40:06.314 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:06.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:40:06.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:40:06.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:06.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-163208757251 2020-03-11T22:40:06.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:06.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:06.696 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-163208757251 2020-03-11T22:40:06.696 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:06.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=163208757251 2020-03-11T22:40:06.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2020-03-11T22:40:06.698 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:06.698 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 163208757251 2020-03-11T22:40:07.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 163208757251 -lt 163208757251 2020-03-11T22:40:07.109 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:07.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:07.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:07.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:07.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:07.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:07.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:07.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:07.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:07.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:07.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:07.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:07.948 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:07.948 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:07.949 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:08.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:08.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:08.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:08.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:08.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:08.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:08.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:08.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:08.529 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:08.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:08.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:08.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:08.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:09.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:09.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:09.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:09.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:09.362 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:09.362 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:09.362 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:09.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:09.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:09.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:09.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:09.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:09.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:09.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:10.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:10.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:10.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:10.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:10.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:10.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:10.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:10.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:10.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:10.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:10.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:10.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:10.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:10.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:10.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:10.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:11.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:11.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:11.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:11.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:11.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:11.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:11.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:11.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:11.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:11.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:11.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:11.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:11.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:12.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:12.138 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:12.138 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:12.138 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:12.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:12.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:12.615 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:12.615 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:13.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:13.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:13.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:13.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:13.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:13.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:13.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:13.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:13.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:13.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:13.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:13.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:13.890 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:14.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:14.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:14.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:14.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:14.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:14.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:14.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:14.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:14.732 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:16.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:16.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:16.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:16.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:16.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:16.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:16.894 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:16.895 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:17.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:17.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:17.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:17.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:17.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:17.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:17.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:17.733 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:18.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:18.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:18.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:18.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:18.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:21.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:21.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:21.409 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:21.409 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:21.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:21.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:21.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:21.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:22.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:22.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:22.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:22.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:22.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:22.262 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:22.262 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:22.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:22.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:22.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:22.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:22.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:22.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:22.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:40:29.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:29.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:29.144 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:29.144 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:29.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:29.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:29.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:29.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:29.512 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:29.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:29.513 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:29.513 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:29.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:29.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:29.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:29.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:29.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:29.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:29.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:29.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:29.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:30.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:30.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:30.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:40:30.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:30.454 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:40:37.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:37.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:37.754 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:37.754 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:37.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:37.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:37.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:37.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:38.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:38.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:38.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:38.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:38.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:38.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:38.598 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:38.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:39.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:39.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:39.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:40:39.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:40:39.536 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:40:39.536 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "18c4aa29-3f0b-4519-aa64-070b9ae52006", 2020-03-11T22:40:39.536 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:40:39.537 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:40:39.537 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:40:39.532805+0000", 2020-03-11T22:40:39.537 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:40:39.537 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:40:39.537 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:40:39.538 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:40:39.539 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.539 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:40:39.539 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:40:39.539 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.539 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:40:39.540 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:39.540 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:40:39.540 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:40:39.540 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:40:39.540 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.541 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:40:39.542 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 32 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.544 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.544 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.544 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.544 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.544 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.545 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.546 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.546 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.546 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.546 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.546 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.547 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.548 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.549 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.550 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.552 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.552 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.552 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.552 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.552 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.556 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.558 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.561 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:40:39.563 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:38:59.012576Z", 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:38:59.012576Z", 2020-03-11T22:40:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:40:39.565 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:40:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:40:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:40:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:40:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:40:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:40:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:40:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:40:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:40:39.570 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 44, 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:38:59.755684+0000", 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:40:06.390727+0000", 2020-03-11T22:40:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:40:03.227128+0000", 2020-03-11T22:40:39.572 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:40:04.879384+0000", 2020-03-11T22:40:39.572 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:40:39.572 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:40:39.572 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:40:39.572 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:40:39.573 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 7, 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:40:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:40:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:39:22.084559+0000", 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 1, 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool", 2020-03-11T22:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:40:39.577 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:40:39.577 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:40:39.577 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 0, 2020-03-11T22:40:39.577 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 32, 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 32, 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 32, 2020-03-11T22:40:39.578 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 32, 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:40:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "43", 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:40:39.580 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "43", 2020-03-11T22:40:39.581 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:40:39.581 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:40:39.581 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 0, 2020-03-11T22:40:39.581 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 0, 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [], 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:40:39.582 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:40:39.583 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:40:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:40:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:40:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:40:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:40:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "", 2020-03-11T22:40:39.585 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:40:39.585 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:40:39.585 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.585 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:40:39.585 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:40:39.586 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:40:39.586 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:40:39.586 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:40:39.586 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:40:39.587 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:40:39.587 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:40:39.587 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 0, 2020-03-11T22:40:39.587 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:40:39.588 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:40:39.588 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:40:39.588 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:40:39.588 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.588 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.589 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:40:39.589 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.589 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:40:39.589 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "7cc010f5-1143-4f6a-94c4-303fedff664a", 2020-03-11T22:40:39.589 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:40:39.590 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:40:39.590 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:40:39.590 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:40:39.590 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:40:39.590 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 31, 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 37, 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 43, 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:40:39.591 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.593 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.593 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:40:39.593 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.593 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.593 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:40:39.594 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.596 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.596 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.597 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.598 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:40:39.598 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.598 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.598 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.599 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.599 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:39.599 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.599 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.599 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:40:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.601 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:40:39.602 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/39352", 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/39352", 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39352", 2020-03-11T22:40:39.603 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39352", 2020-03-11T22:40:39.604 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:40:39.604 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:40:39.604 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "f9d3fe39-2937-4660-9a35-ada2db8fe0c4", 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:40:39.605 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 23, 2020-03-11T22:40:39.606 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 33, 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.607 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.608 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.610 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.611 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.612 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.613 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.614 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/34889", 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/34889", 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34889", 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34889", 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.615 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "ad4265b4-f9e4-4561-9f43-b939c3dfabe7", 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:40:39.616 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 31, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 38, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 34, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:40:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.619 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:39.619 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.619 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.619 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.621 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.621 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:40:39.621 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.621 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.622 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.622 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:39.622 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.623 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.623 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.623 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.623 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:39.624 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.624 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.624 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.624 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.624 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.625 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:39.627 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.628 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/39546", 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/39546", 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39546", 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39546", 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:40:39.629 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:40:39.630 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:56.898801+0000", 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:07.000134+0000", 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:40:39.631 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:57.416201+0000", 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:40:39.632 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:13.370724+0000", 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:57.935018+0000", 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:40:39.633 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:19.820428+0000", 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:40:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:40:39.635 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:40:39.636 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:40:39.637 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/39352,v1:127.0.0.1:6805/39352]", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:39.638 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:39.639 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:40:39.640 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/39352,v1:127.0.0.1:6803/39352]", 2020-03-11T22:40:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39352,v1:127.0.0.1:6809/39352]", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39352,v1:127.0.0.1:6807/39352]", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:39.643 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:40:39.644 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/34889,v1:127.0.0.1:6813/34889]", 2020-03-11T22:40:39.645 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:39.646 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:40:39.647 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:40:39.648 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/34889,v1:127.0.0.1:6811/34889]", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34889,v1:127.0.0.1:6817/34889]", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34889,v1:127.0.0.1:6815/34889]", 2020-03-11T22:40:39.649 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:39.650 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:40:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/39546,v1:127.0.0.1:6813/39546]", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:39.652 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:39.653 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:40:39.654 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/39546,v1:127.0.0.1:6811/39546]", 2020-03-11T22:40:39.655 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39546,v1:127.0.0.1:6817/39546]", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39546,v1:127.0.0.1:6815/39546]", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:39.656 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:40:39.657 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 44, 2020-03-11T22:40:39.658 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.659 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:40:39.660 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:40:39.661 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:40:39.662 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.662 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.662 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:40:39.662 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:40:39.662 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.663 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.663 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:40:39.663 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:40:39.663 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.664 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.664 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:40:39.664 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:40:39.664 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.664 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.665 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:40:39.665 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:40:39.665 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.665 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.665 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:40:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:40:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:40:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:40:39.667 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.667 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.667 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:40:39.667 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:40:39.667 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:40:39.668 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.669 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.670 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:40:39.671 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.672 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.673 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:40:39.674 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:40:39.675 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:40:39.676 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:40:39.677 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.678 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:40:39.679 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.680 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:40:39.681 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:40:39.682 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:40:39.682 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.683 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:40:39.684 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:40:39.685 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.685 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:40:39.685 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:40:39.685 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:40:39.685 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.686 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:40:39.687 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.687 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:40:39.687 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.687 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.687 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:40:39.688 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:40:39.689 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 0, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:40:39.690 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:40:39.691 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:40:39.692 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:40:39.693 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:40:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:40:39.695 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 32, 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 32, 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:40:39.696 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:40:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:40:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:40:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:40:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:40:39.698 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:39.698 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:39.698 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:39.698 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:39.698 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:40:39.699 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:40:39.699 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:40:39.699 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:40:39.699 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:40:39.700 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:39.700 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:39.700 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:39.700 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:39.701 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:39.701 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:39.701 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:40:39.701 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:40:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:40:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:40:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:40:39.703 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:40:39.703 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:39.703 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:39.703 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:39.704 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:39.704 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:39.704 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:39.704 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:39.705 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:39.705 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:39.705 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:40:39.705 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:40:39.706 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.706 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:40:39.706 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:40:39.706 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:40:39.706 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:40:39.707 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:40:39.707 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:40:39.707 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:40:39.707 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:39.708 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:40:39.708 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:40:39.708 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:40:39.708 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.709 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 512, 2020-03-11T22:40:39.709 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:39.709 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 64, 2020-03-11T22:40:39.709 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 64, 2020-03-11T22:40:39.710 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:40:39.710 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.710 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:40:39.710 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:40:39.711 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:40:39.711 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 65, 2020-03-11T22:40:39.711 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:40:39.711 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:39.711 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:39.712 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:40:39.712 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3180672, 2020-03-11T22:40:39.712 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 34624, 2020-03-11T22:40:39.712 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 6, 2020-03-11T22:40:39.713 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145721, 2020-03-11T22:40:39.713 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392128, 2020-03-11T22:40:39.713 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:40:39.713 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318865539072, 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 35454976, 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 438648, 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 7102, 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221218370 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:40:39.715 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.716 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:40:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 65, 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:40:39.718 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3180672, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 34624, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 6, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145721, 2020-03-11T22:40:39.719 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392128, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318865539072, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 35454976, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 438648, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:40:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 7102, 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221218370 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:39.721 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:40:39.722 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.723 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.724 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:40:39.724 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.725 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 32, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48300, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:40:39.726 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:40:39.727 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 100, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 100, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:39.728 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:39.730 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:40:39.731 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:40:39.732 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19529728, 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 143228, 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:40:39.736 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.736 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 512, 2020-03-11T22:40:39.736 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:39.736 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 64, 2020-03-11T22:40:39.737 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 64, 2020-03-11T22:40:39.737 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:40:39.737 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.738 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.738 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:40:39.738 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.738 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:40:39.739 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 158913789962 2020-03-11T22:40:39.739 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.739 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.739 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:40:39.740 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672975 2020-03-11T22:40:39.740 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.740 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.740 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:40:39.741 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 163208757257 2020-03-11T22:40:39.741 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.741 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.741 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:40:39.742 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.742 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:40:39.742 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 32 2020-03-11T22:40:39.742 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.742 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 32, 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:39.743 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.744 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 32, 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 32, 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.745 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.746 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:38:59.756007+0000", 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:40:39.747 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 309, 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:40:39.748 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:40:39.750 INFO:tasks.workunit.client.0.smithi045.stderr:report 3055416292 2020-03-11T22:40:39.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:40:39.750 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:40:39.750 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:40:39.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:40:39.751 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:39.751 INFO:tasks.workunit.client.0.smithi045.stdout:writing non-divergent object existing_4 2020-03-11T22:40:39.887 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.888 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.890 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.891 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.893 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:40:39.894 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:40:39.894 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:39.894 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:40:39.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:40:39.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:39.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:39.965 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:39.965 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:39.965 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:39.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:39.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:40.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:40.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:40.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:40.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:40.024 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:40.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:40.393 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:40:40.393 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:40:40.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:40.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.394 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:40.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789964 2020-03-11T22:40:40.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789964' 2020-03-11T22:40:40.541 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:40.637 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:40.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:40.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789964 1-' 2020-03-11T22:40:40.646 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.646 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:40.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757260 2020-03-11T22:40:40.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789964 1- 2-163208757260' 2020-03-11T22:40:40.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:40.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-158913789964 2020-03-11T22:40:40.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:40.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:40.784 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-158913789964 2020-03-11T22:40:40.784 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:40.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789964 2020-03-11T22:40:40.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 158913789964' 2020-03-11T22:40:40.785 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:40.785 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 158913789964 2020-03-11T22:40:41.147 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789962 -lt 158913789964 2020-03-11T22:40:41.147 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:42.148 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:42.149 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:42.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789965 -lt 158913789964 2020-03-11T22:40:42.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:42.524 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:42.524 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:42.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:42.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:42.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:42.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:42.526 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:42.527 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:42.527 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:40:42.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:40:42.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:40:42.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:42.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-163208757260 2020-03-11T22:40:42.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:42.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:42.890 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-163208757260 2020-03-11T22:40:42.890 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:42.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=163208757260 2020-03-11T22:40:42.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2020-03-11T22:40:42.891 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 163208757260 2020-03-11T22:40:42.892 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:43.286 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 163208757260 -lt 163208757260 2020-03-11T22:40:43.286 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:43.287 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:43.287 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:43.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:40:43.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:43.771 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:43.771 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:43.771 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:43.772 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:43.772 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:43.772 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:44.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:44.131 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:44.131 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:44.132 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:44.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:44.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:44.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:44.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:44.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:44.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:44.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:45.057 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:45.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:45.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:45.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:45.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:45.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:45.544 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:46.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:46.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:46.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:46.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:46.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:46.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:46.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:46.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:46.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:46.214 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:46.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:46.571 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:46.571 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:46.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:47.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:47.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:47.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:47.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:47.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:47.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:47.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:47.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:47.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:47.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:47.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:47.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:47.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:47.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:47.916 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:47.916 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:47.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:47.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:47.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:47.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:47.917 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:47.918 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:48.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:48.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:48.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:48.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:48.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:48.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:48.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:48.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:48.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:48.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:48.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:48.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:48.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:49.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:50.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:50.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:50.028 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:50.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:50.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:50.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:50.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:50.029 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:50.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:50.396 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:50.397 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:50.397 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:50.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:50.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:50.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:50.881 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:51.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:51.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:51.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:51.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:51.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:52.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:52.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:52.958 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:52.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:52.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:52.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:52.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:52.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:53.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:53.331 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:53.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:53.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:53.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:53.808 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:53.809 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:54.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:54.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:54.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:54.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:54.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:57.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:57.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:57.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:57.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:57.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:57.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:57.480 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:57.480 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:57.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:57.844 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:57.844 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:57.844 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:58.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:58.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:58.315 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:58.315 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:05.213 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:05.213 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:05.214 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:05.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:05.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:05.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:05.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:05.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:05.580 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:05.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:05.581 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:05.582 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:06.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:06.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:06.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:06.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:06.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:06.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:06.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:06.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:06.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:41:13.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:13.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:13.830 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:13.830 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:13.830 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:13.831 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:13.831 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:13.831 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:14.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:14.202 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:14.202 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:14.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:14.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:14.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:14.672 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:15.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:15.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:15.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:41:15.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:41:15.664 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:41:15.665 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "18c4aa29-3f0b-4519-aa64-070b9ae52006", 2020-03-11T22:41:15.665 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:41:15.665 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:41:15.665 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:41:15.661494+0000", 2020-03-11T22:41:15.665 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:41:15.666 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:15.667 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:41:15.668 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:41:15.669 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 64 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.670 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.2 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.3 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.671 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.4 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.5 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.672 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.6 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.7 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.8 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.9 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.a is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.674 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.b is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.c is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.675 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.d is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.e is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.f is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.676 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.10 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.11 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.677 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.12 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.13 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.678 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.14 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.15 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.16 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.17 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.18 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.680 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.19 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1a is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1b is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.681 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1c is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1d is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.682 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1e is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.1f is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:41:15.683 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:41:15.684 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.685 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:41:15.686 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:38:59.012576Z", 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:38:59.012576Z", 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:41:15.687 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:41:15.688 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:41:15.689 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:41:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:41:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:41:15.691 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.692 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 44, 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:38:59.755684+0000", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:40:06.390727+0000", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:40:03.227128+0000", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:40:04.879384+0000", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:41:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:41:15.694 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 7, 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:41:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:41:15.696 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.697 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:41:15.697 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:41:15.697 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:39:22.084559+0000", 2020-03-11T22:41:15.697 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 1, 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool", 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 0, 2020-03-11T22:41:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 32, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 32, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 32, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 32, 2020-03-11T22:41:15.699 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "43", 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "43", 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 0, 2020-03-11T22:41:15.701 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 0, 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [], 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:41:15.702 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:41:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "", 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.704 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:41:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:41:15.706 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:41:15.706 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 0, 2020-03-11T22:41:15.706 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:41:15.706 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:41:15.707 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:41:15.707 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:41:15.707 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.707 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:41:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:41:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "7cc010f5-1143-4f6a-94c4-303fedff664a", 2020-03-11T22:41:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:41:15.709 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:41:15.709 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:41:15.709 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:41:15.709 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:41:15.710 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 31, 2020-03-11T22:41:15.710 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 37, 2020-03-11T22:41:15.710 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 43, 2020-03-11T22:41:15.710 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:41:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:41:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:41:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.711 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.712 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:41:15.712 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.712 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.712 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.713 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.713 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:41:15.713 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.713 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.714 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.714 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.714 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:41:15.714 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.714 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.715 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.715 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:41:15.715 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.715 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.716 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.716 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.716 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:41:15.716 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.717 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.717 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.717 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.717 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:15.717 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.718 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.718 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.718 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:41:15.718 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.719 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.719 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.719 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.719 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:41:15.720 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.720 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.720 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.720 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.720 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:15.721 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.721 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.721 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.721 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:41:15.722 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.722 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.722 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.722 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.722 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:41:15.723 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39352 2020-03-11T22:41:15.723 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.723 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.723 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.724 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/39352", 2020-03-11T22:41:15.724 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/39352", 2020-03-11T22:41:15.724 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39352", 2020-03-11T22:41:15.724 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39352", 2020-03-11T22:41:15.725 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:41:15.725 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:41:15.725 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:41:15.725 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.726 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.726 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.726 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:41:15.726 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "f9d3fe39-2937-4660-9a35-ada2db8fe0c4", 2020-03-11T22:41:15.726 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:41:15.727 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:41:15.727 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:41:15.727 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:41:15.727 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:41:15.727 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:41:15.728 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:41:15.728 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 23, 2020-03-11T22:41:15.728 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 33, 2020-03-11T22:41:15.728 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:41:15.729 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:41:15.729 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.729 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.729 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.730 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:15.730 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.730 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.730 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.730 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.731 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:15.731 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.731 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.731 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.732 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.732 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:41:15.732 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.732 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.734 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.734 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:15.734 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.734 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.735 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.735 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.735 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:15.735 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.735 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.736 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.736 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:15.736 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.736 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.737 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.737 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.737 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:15.737 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.738 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.738 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.738 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.738 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:15.738 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.739 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.739 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.739 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:15.739 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.740 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.740 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.740 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.740 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:15.740 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 34889 2020-03-11T22:41:15.741 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.741 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.741 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.741 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/34889", 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/34889", 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34889", 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34889", 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:41:15.744 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "ad4265b4-f9e4-4561-9f43-b939c3dfabe7", 2020-03-11T22:41:15.744 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:41:15.744 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:41:15.744 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:41:15.744 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:41:15.745 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:41:15.745 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 31, 2020-03-11T22:41:15.745 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 38, 2020-03-11T22:41:15.745 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:41:15.746 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 34, 2020-03-11T22:41:15.746 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:41:15.746 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:41:15.746 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.747 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.747 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.747 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:15.747 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.748 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.748 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.748 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.748 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:15.748 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.749 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.749 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.749 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.749 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:41:15.750 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.750 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.750 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.750 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:15.751 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.751 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.751 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.751 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.752 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:15.752 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.752 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.752 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.752 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.753 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:15.753 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.753 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.753 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.754 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:15.754 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.754 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.754 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.755 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.755 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:15.755 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.755 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.755 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.756 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.756 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:15.756 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:41:15.756 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.757 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:41:15.757 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:15.757 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.757 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.757 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.758 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:41:15.758 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:15.758 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 39546 2020-03-11T22:41:15.758 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.759 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.759 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.759 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/39546", 2020-03-11T22:41:15.759 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/39546", 2020-03-11T22:41:15.760 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39546", 2020-03-11T22:41:15.760 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39546", 2020-03-11T22:41:15.760 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:41:15.760 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:41:15.760 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:41:15.761 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.761 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.761 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.761 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:41:15.762 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.762 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:41:15.762 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:56.898801+0000", 2020-03-11T22:41:15.762 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:41:15.763 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:41:15.763 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:41:15.763 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:41:15.763 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:07.000134+0000", 2020-03-11T22:41:15.763 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:57.416201+0000", 2020-03-11T22:41:15.765 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:41:15.765 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:41:15.765 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:41:15.765 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:41:15.766 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:13.370724+0000", 2020-03-11T22:41:15.766 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:41:15.766 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.766 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.767 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:41:15.767 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:39:57.935018+0000", 2020-03-11T22:41:15.767 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:41:15.767 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:41:15.767 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:41:15.768 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:41:15.768 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:19.820428+0000", 2020-03-11T22:41:15.768 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:41:15.768 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.769 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.769 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:41:15.769 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:41:15.769 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:41:15.770 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:41:15.770 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:41:15.770 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:41:15.770 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.771 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.772 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/39352,v1:127.0.0.1:6805/39352]", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:15.773 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:15.774 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:41:15.775 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/39352,v1:127.0.0.1:6803/39352]", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:41:15.776 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39352,v1:127.0.0.1:6809/39352]", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39352,v1:127.0.0.1:6807/39352]", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.778 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/34889,v1:127.0.0.1:6813/34889]", 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:15.779 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:15.780 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:15.781 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/34889,v1:127.0.0.1:6811/34889]", 2020-03-11T22:41:15.782 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34889,v1:127.0.0.1:6817/34889]", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34889,v1:127.0.0.1:6815/34889]", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:15.783 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:41:15.784 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/39546,v1:127.0.0.1:6813/39546]", 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:15.785 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:15.786 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/39546,v1:127.0.0.1:6811/39546]", 2020-03-11T22:41:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39546,v1:127.0.0.1:6817/39546]", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39546,v1:127.0.0.1:6815/39546]", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:15.790 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 44, 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.791 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:41:15.792 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:41:15.794 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:41:15.795 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:41:15.796 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:41:15.797 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:41:15.798 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:41:15.799 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:41:15.800 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:41:15.803 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.804 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:41:15.805 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:41:15.806 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.807 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:41:15.808 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:41:15.809 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.810 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:41:15.811 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.814 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:41:15.815 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:41:15.816 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 0, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:41:15.817 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:41:15.818 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:41:15.819 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:41:15.820 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:41:15.821 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 32, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 32, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:41:15.822 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 47947, 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:41:15.823 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:41:15.824 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:15.825 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:41:15.826 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:15.827 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:41:15.828 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:41:15.829 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 513, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 64, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 64, 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:41:15.830 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 65, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:15.831 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3180672, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 34624, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 6, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145721, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392128, 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:41:15.832 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318865539072, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 35454976, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 437942, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:41:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 7102, 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221218370 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:15.834 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:15.835 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:41:15.836 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 65, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3180672, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 34624, 2020-03-11T22:41:15.837 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 6, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145721, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392128, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318865539072, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 35454976, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 437942, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 7102, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221218370 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.839 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.840 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:41:15.841 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 32, 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:41:15.842 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 47947, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 100, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 0, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 300, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:15.843 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:41:15.844 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:15.845 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:15.846 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19529728, 2020-03-11T22:41:15.848 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 142522, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 513, 2020-03-11T22:41:15.849 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 64, 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 64, 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.850 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 158913789971 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672975 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.851 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 163208757267 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:41:15.852 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 32 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 32, 2020-03-11T22:41:15.853 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.854 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 32, 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 32, 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:41:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:41:15.856 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:38:59.756007+0000", 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 390, 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:41:15.857 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:41:15.858 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:41:15.858 INFO:tasks.workunit.client.0.smithi045.stderr:report 3804107845 2020-03-11T22:41:15.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:41:15.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:41:15.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:15.859 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery 2020-03-11T22:41:16.022 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:16.022 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.022 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.022 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.023 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.024 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.025 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 44'101 44:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.026 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.027 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:16.027 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:16.027 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:16.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:16.032 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:16.033 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:41:16.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:16.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:41:16.129 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:16.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:16.139 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:41:16.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:16.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:41:16.235 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:16.242 INFO:tasks.workunit.client.0.smithi045.stdout:revive divergent 1 2020-03-11T22:41:16.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:41:16.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:16.584 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:16.584 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.585 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.586 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.587 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.588 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.589 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.589 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 44'101 44:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.589 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:16.589 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:16.589 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:16.590 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:16.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:41:17.302 INFO:tasks.workunit.client.0.smithi045.stderr:noup is set 2020-03-11T22:41:17.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:17.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:17.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:17.315 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:17.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:17.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:17.316 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:17.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:17.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:17.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:17.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:17.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:17.319 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:41:17.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:17.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:17.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:17.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:17.341 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:17.338+0000 7fc04f586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:17.342 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:17.340+0000 7fc04f586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:17.345 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:17.342+0000 7fc04f586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:17.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:41:17.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:41:17.860 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:17.857+0000 7fc04f586ec0 -1 Falling back to public interface 2020-03-11T22:41:18.753 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:18.750+0000 7fc04f586ec0 -1 osd.1 31 log_to_monitors {default=true} 2020-03-11T22:41:22.681 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery divergent 2020-03-11T22:41:22.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:41:22.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:23.036 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:23.036 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.037 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.038 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.039 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 45:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.040 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 44'101 45:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:23.041 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:23.042 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:23.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:41:23.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:23.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:41:23.144 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:23.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:41:24.041 INFO:tasks.workunit.client.0.smithi045.stderr:noup is unset 2020-03-11T22:41:24.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:41:24.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:24.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:24.052 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:41:24.434 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 37 up_thru 43 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/39352,v1:127.0.0.1:6803/39352] [v2:127.0.0.1:6804/39352,v1:127.0.0.1:6805/39352] exists,up 7cc010f5-1143-4f6a-94c4-303fedff664a 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:24.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:24.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42071,v1:127.0.0.1:6819/42071] [v2:127.0.0.1:6820/42071,v1:127.0.0.1:6821/42071] exists,up f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:25.201 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6810/39546,v1:127.0.0.1:6811/39546] [v2:127.0.0.1:6812/39546,v1:127.0.0.1:6813/39546] exists,up ad4265b4-f9e4-4561-9f43-b939c3dfabe7 2020-03-11T22:41:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:25.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:25.543 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:25.543 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.543 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.544 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.545 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 45:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.546 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.548 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 44'101 45:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.548 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.548 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:25.548 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:25.549 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:25.554 INFO:tasks.workunit.client.0.smithi045.stdout:wait for peering 2020-03-11T22:41:25.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:41:25.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:25.896 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:25.897 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622424+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.897 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620533+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.897 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620980+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.897 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.622045+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.898 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620625+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.898 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616964+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.898 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617908+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.898 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620239+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.899 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621534+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.899 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620131+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.899 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619705+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.900 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618311+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.900 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619359+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.900 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620963+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.900 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619268+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.901 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.620495+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.901 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619175+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.901 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617922+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.901 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617903+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.618023+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:21.336671+0000 31'100 45:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621387+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621950+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617027+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616937+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.616917+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.621641+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617795+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.617544+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.904 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619151+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.904 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:06.617444+0000 44'101 45:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.904 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:06.619832+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:25.904 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:25.904 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:25.905 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:25.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:41:34.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:41:34.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:34.643 INFO:tasks.workunit.client.0.smithi045.stdout:killing divergent 1 2020-03-11T22:41:34.997 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:34.997 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450487+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.450942+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.452619+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.452063+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.449287+0000 31'100 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.406576+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.998 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.408020+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.452129+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451334+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.448695+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.407536+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.407555+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.451758+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:34.999 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450890+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.448304+0000 31'100 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.449146+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.407207+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451351+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.450093+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.448167+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.000 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.451248+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:24.449478+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450084+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.405707+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.449847+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:24.407360+0000 31'100 50:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451853+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.406849+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:24.406601+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:24.450426+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.450924+0000 44'101 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450487+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:35.003 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:35.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:41:35.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:35.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:35.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:35.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:35.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:35.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:35.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17438.out 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17438.out 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:35.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17438.out 2020-03-11T22:41:35.758 INFO:tasks.workunit.client.0.smithi045.stderr:Exporting 1.0 info 1.0( v 31'100 (31'90,31'100] local-lis/les=48/50 n=3 ec=23/23 lis/c=48/48 les/c/f=50/50/0 sis=48 pruub=10.168448712s) 2020-03-11T22:41:35.798 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:35.819 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:35.831 INFO:tasks.workunit.client.0.smithi045.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:35.835 INFO:tasks.workunit.client.0.smithi045.stderr:Export successful 2020-03-11T22:41:35.836 INFO:tasks.workunit.client.0.smithi045.stdout: marking collection for removal 2020-03-11T22:41:35.836 INFO:tasks.workunit.client.0.smithi045.stdout:setting '_remove' omap key 2020-03-11T22:41:35.836 INFO:tasks.workunit.client.0.smithi045.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:41:35.861 INFO:tasks.workunit.client.0.smithi045.stderr:Remove successful 2020-03-11T22:41:36.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17438.out 2020-03-11T22:41:36.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:36.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:36.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:36.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:36.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:36.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17438.out 2020-03-11T22:41:36.846 INFO:tasks.workunit.client.0.smithi045.stdout:get_pg_num_history pg_num_history pg_num_history(e50 pg_nums {1={23=1,43=32}} deleted_pools ) 2020-03-11T22:41:36.847 INFO:tasks.workunit.client.0.smithi045.stdout:Importing pgid 1.0 2020-03-11T22:41:36.847 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:36.847 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:41:36.886 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:36.886 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:41:36.895 INFO:tasks.workunit.client.0.smithi045.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:36.895 INFO:tasks.workunit.client.0.smithi045.stdout:snapset 0=[]:{} 2020-03-11T22:41:36.903 INFO:tasks.workunit.client.0.smithi045.stdout:write_pg epoch 50 info 1.0( v 31'100 (31'90,31'100] local-lis/les=48/50 n=3 ec=23/23 lis/c=48/48 les/c/f=50/50/0 sis=48 pruub=10.168448712s) 2020-03-11T22:41:36.911 INFO:tasks.workunit.client.0.smithi045.stdout:Import successful 2020-03-11T22:41:37.428 INFO:tasks.workunit.client.0.smithi045.stdout:reviving divergent 1 2020-03-11T22:41:37.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:41:37.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:37.781 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:37.781 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450487+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.450942+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.452619+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.452063+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.449287+0000 31'100 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.406576+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.408020+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.782 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.452129+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451334+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.448695+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.407536+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.407555+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.451758+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450890+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.783 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.448304+0000 31'100 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.449146+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.407207+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451351+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.450093+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.448167+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.451248+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.784 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:24.449478+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450084+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.405707+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.449847+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:24.407360+0000 31'100 50:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.451853+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:24.406849+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.785 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:24.406601+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:24.450426+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:41:24.450924+0000 44'101 50:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:24.450487+0000 31'100 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:37.786 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:37.787 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:37.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:37.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:37.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:37.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:37.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:37.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:37.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:37.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:37.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:37.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:37.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:37.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:37.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:37.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:37.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:37.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:37.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:37.799 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:41:37.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:37.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:37.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:37.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:37.822 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:37.818+0000 7fb759e82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:37.823 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:37.820+0000 7fb759e82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:37.825 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:37.821+0000 7fb759e82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:38.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:41:38.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:38.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:38.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:38.201 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:41:38.344 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:41:38.337+0000 7fb759e82ec0 -1 Falling back to public interface 2020-03-11T22:41:38.566 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42071,v1:127.0.0.1:6819/42071] [v2:127.0.0.1:6820/42071,v1:127.0.0.1:6821/42071] exists,up f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:41:38.566 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:41:38.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:38.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:38.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:38.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:38.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:38.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:38.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:38.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:38.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:38.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:38.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:38.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:38.977 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42071,v1:127.0.0.1:6819/42071] [v2:127.0.0.1:6820/42071,v1:127.0.0.1:6821/42071] exists,up f9d3fe39-2937-4660-9a35-ada2db8fe0c4 2020-03-11T22:41:38.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:38.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:38.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:38.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:41:58.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:41:58.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:58.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:58.967 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:58.967 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:58.968 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:41:58.968 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:41:58.968 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:58.968 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:41:59.071 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:41:59.071 INFO:tasks.workunit.client.0.smithi045.stdout: "ops": [], 2020-03-11T22:41:59.071 INFO:tasks.workunit.client.0.smithi045.stdout: "num_ops": 0 2020-03-11T22:41:59.071 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:41:59.079 INFO:tasks.workunit.client.0.smithi045.stdout:allowing recovery 2020-03-11T22:41:59.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:41:59.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:59.424 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:59.424 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.726563+0000 31'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.424 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.946559+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.425 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599331+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.425 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:39.996809+0000 31'100 59:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.425 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:50.725441+0000 31'100 59:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.425 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.946716+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:41:43.141764+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.426 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:51.914705+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:51.914664+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.426 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.997340+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.426 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.726284+0000 31'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.426 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.945944+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.427 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:41:57.982926+0000 31'100 63:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:57.982878+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.427 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.996811+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.427 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:50.725566+0000 31'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.427 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.945893+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.428 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599155+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.428 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.994994+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.428 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:41:55.972907+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:55.972856+0000 31'100 2020-03-11T22:41:55.972856+0000 0 2020-03-11T22:41:59.428 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:39.996765+0000 31'100 59:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.429 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.726031+0000 31'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.429 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:41:55.505821+0000 31'100 61:151 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:41:55.505776+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.429 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.998160+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.430 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:46.600162+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.430 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:48.947011+0000 50'101 60:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.430 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:50.630567+0000 31'100 59:154 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:41:46.146156+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.431 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:39.992334+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.431 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:41:54.925498+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:54.925440+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.431 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:48.946852+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.431 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.726192+0000 31'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.432 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:41:39.994098+0000 31'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.432 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599307+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.432 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:41:48.946012+0000 44'101 60:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.432 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.630451+0000 31'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:41:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:41:59.433 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:59.434 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:41:59.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:59.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:41:59.561 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:59.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:59.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:41:59.697 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:59.703 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:59.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:41:59.831 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:59.839 INFO:tasks.workunit.client.0.smithi045.stdout:reading divergent objects 2020-03-11T22:41:59.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:41:59.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:00.181 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:00.181 INFO:tasks.workunit.client.0.smithi045.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.726563+0000 31'100 63:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:56.546510+0000 31'100 63:155 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599331+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.1c 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:55.257619+0000 31'100 63:153 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:55.257571+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:50.725441+0000 31'100 63:159 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.946716+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:41:43.141764+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:51.914705+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:51.914664+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.182 INFO:tasks.workunit.client.0.smithi045.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.997340+0000 31'100 63:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.726284+0000 31'100 63:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.595539+0000 31'100 63:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:41:57.982926+0000 31'100 63:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:57.982878+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.996811+0000 31'100 63:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:50.725566+0000 31'100 63:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:48.945893+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.183 INFO:tasks.workunit.client.0.smithi045.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599155+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.994994+0000 31'100 63:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:41:55.972907+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:55.972856+0000 31'100 2020-03-11T22:41:55.972856+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:54.210449+0000 31'100 63:153 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:54.210397+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.726031+0000 31'100 63:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:41:55.505821+0000 31'100 61:151 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:41:55.505776+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:39.998160+0000 31'100 63:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.184 INFO:tasks.workunit.client.0.smithi045.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:46.600162+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:56.595400+0000 50'101 63:155 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:50.630567+0000 31'100 63:158 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:41:46.146156+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:58.261418+0000 31'100 63:154 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:58.261351+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:41:54.925498+0000 31'100 62:134 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:54.925440+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:48.946852+0000 31'100 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.726192+0000 31'100 63:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.185 INFO:tasks.workunit.client.0.smithi045.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:41:39.994098+0000 31'100 63:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.186 INFO:tasks.workunit.client.0.smithi045.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:46.599307+0000 31'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.186 INFO:tasks.workunit.client.0.smithi045.stdout:1.e 3 0 0 0 0 1156 0 0 14 14 active+clean 2020-03-11T22:41:56.595379+0000 44'101 63:157 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.186 INFO:tasks.workunit.client.0.smithi045.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.630451+0000 31'100 63:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:22.134727+0000 0'0 2020-03-11T22:39:22.134727+0000 0 2020-03-11T22:42:00.186 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:42:00.186 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:00.187 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:42:00.191 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:42:00.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:42:00.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:00.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:42:00.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:00.272 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:42:00.308 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:00.308 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:42:00.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:00.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:42:00.428 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:42:00.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:00.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:42:00.461 INFO:tasks.workunit.client.0.smithi045.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:42:00.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:00.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:42:00.569 INFO:tasks.workunit.client.0.smithi045.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:42:00.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:42:00.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:00.573 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4532.0:1 2020-03-11T22:39:46.149688+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 20 _merge_object_divergent_entries: keeping 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4532.0:1 2020-03-11T22:39:46.149688+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 31'102 last_divergent_update: 31'102 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:42:00.581 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 31'106 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4532.0:1 2020-03-11T22:39:46.149688+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.884+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'89 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:39:46.160606+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 20 _merge_object_divergent_entries: keeping 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:39:46.160606+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 31'106 last_divergent_update: 31'106 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: more recent entry found: 44'101 (31'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:40:39.922704+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:39:46.154231+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 20 _merge_object_divergent_entries: keeping 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:39:46.154231+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 31'103 last_divergent_update: 31'103 2020-03-11T22:42:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.969+0000 7fc02b73d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:39:46.159401+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: keeping 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:39:46.159401+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 31'105 last_divergent_update: 31'105 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:42:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 31'106 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:39:46.159401+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'84 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:39:46.156794+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: keeping 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:39:46.156794+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 31'104 last_divergent_update: 31'104 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 31'106 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:39:46.156794+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'90 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 31'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:39:46.138728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 20 _merge_object_divergent_entries: keeping 31'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:39:46.138728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 31'101 last_divergent_update: 31'101 2020-03-11T22:42:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:42:00.590 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:23.970+0000 7fc02c73f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:00.592 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:00.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:00.597 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:00.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:00.605 INFO:tasks.workunit.client.0.smithi045.stdout:success 2020-03-11T22:42:00.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:42:00.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17438.out 2020-03-11T22:42:00.606 INFO:tasks.workunit.client.0.smithi045.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17438.out': No such file or directory 2020-03-11T22:42:00.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:42:00.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:42:00.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:42:01.247 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' does not exist 2020-03-11T22:42:01.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:42:01.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:01.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:01.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:01.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:01.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:01.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:01.365 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:01.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:01.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:01.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:01.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:01.370 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:01.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:01.371 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:01.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:01.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:01.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:01.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:01.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:01.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:01.374 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:01.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:01.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:01.376 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:01.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:01.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:42:01.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:01.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:01.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:01.396 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:01.397 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:01.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:01.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:01.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:01.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:01.398 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:01.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:01.399 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:01.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:01.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:01.400 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:01.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:01.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:01.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:01.401 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:01.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:01.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:01.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:01.404 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:01.404 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.404 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:42:01.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:01.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:01.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:42:01.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:42:01.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17438 2020-03-11T22:42:01.407 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:42:01.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:42:01.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:42:01.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:42:01.410 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:42:01.411 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:42:01.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:42:01.411 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:42:01.411 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:42:01.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:42:01.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:42:01.710 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.711 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:01.712 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:42:01.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:42:01.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:42:01.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:01.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:01.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:42:01.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:42:01.865 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:01.865 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:01.865 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:01.865 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:01.866 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.866 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.866 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:42:01.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:01.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get fsid 2020-03-11T22:42:01.955 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:42:01.955 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:01.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:01.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:42:01.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:01.958 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-mon.a.asok 2020-03-11T22:42:01.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:01.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17438/ceph-mon.a.asok config get mon_host 2020-03-11T22:42:02.047 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:42:02.047 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:42:02.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:42:02.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:42:02.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:42:02.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:42:02.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:42:02.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:42:02.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:02.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:02.965 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:02.965 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:02.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:02.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:02.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:42:02.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:02.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:02.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:42:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:42:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:02.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:02.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:02.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:02.998 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:02.998 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:02.999 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:02.999 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:02.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:02.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:02.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:03.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:03.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:03.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:03.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:42:03.001 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:03.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:42:03.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 ced68667-c2e2-48d3-8915-2f9a40be4ea1' 2020-03-11T22:42:03.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:03.003 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:42:03.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA7aWleKjqwABAATQtFyzwOm+qhpuZbA4iQIg== 2020-03-11T22:42:03.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA7aWleKjqwABAATQtFyzwOm+qhpuZbA4iQIg=="}' 2020-03-11T22:42:03.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ced68667-c2e2-48d3-8915-2f9a40be4ea1 -i td/divergent-priors/0/new.json 2020-03-11T22:42:03.362 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:42:03.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:42:03.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA7aWleKjqwABAATQtFyzwOm+qhpuZbA4iQIg== --osd-uuid ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:42:03.402 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.400+0000 7f7ea022dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:03.404 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.402+0000 7f7ea022dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:03.406 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.404+0000 7f7ea022dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:03.407 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.404+0000 7f7ea022dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:03.407 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.404+0000 7f7ea022dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:03.407 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.404+0000 7f7ea022dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:03.452 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:03.450+0000 7f7ea022dec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:42:06.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:42:06.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:06.598 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:42:06.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:42:06.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:07.078 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:42:07.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:42:07.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:07.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:07.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:07.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:07.104+0000 7f2b51ad1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:07.108 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:07.106+0000 7f2b51ad1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:07.110 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:07.107+0000 7f2b51ad1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:07.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:42:07.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:07.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:07.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:07.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:07.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:07.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:07.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:07.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:07.444 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:42:07.626 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:07.624+0000 7f2b51ad1ec0 -1 Falling back to public interface 2020-03-11T22:42:07.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:08.403 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:08.400+0000 7f2b51ad1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:42:08.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:08.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:08.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:08.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:08.827 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:42:08.828 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:09.207 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43569,v1:127.0.0.1:6803/43569] [v2:127.0.0.1:6804/43569,v1:127.0.0.1:6805/43569] exists,up ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:42:09.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:09.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:09.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:09.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:09.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:09.210 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:09.211 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:09.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:09.213 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:09.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:09.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:42:09.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 5b96c0dd-0926-4a5f-acff-35ff3d1763e2' 2020-03-11T22:42:09.214 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:09.214 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:42:09.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBBaWleKkxaDRAA60l94SxH4wncLEDjJPtW8Q== 2020-03-11T22:42:09.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBBaWleKkxaDRAA60l94SxH4wncLEDjJPtW8Q=="}' 2020-03-11T22:42:09.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 -i td/divergent-priors/1/new.json 2020-03-11T22:42:09.796 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:42:09.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:42:09.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBBaWleKkxaDRAA60l94SxH4wncLEDjJPtW8Q== --osd-uuid 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:42:09.837 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.834+0000 7f014353eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:09.842 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.840+0000 7f014353eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:09.844 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.842+0000 7f014353eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:09.845 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.842+0000 7f014353eec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:09.845 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.842+0000 7f014353eec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:09.845 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.842+0000 7f014353eec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:09.942 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:09.940+0000 7f014353eec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:42:13.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:42:13.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:13.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:42:13.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:13.156 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:42:13.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:42:13.642 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:42:13.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:13.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:13.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:13.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:13.671 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:13.668+0000 7f23c11d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:13.672 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:13.670+0000 7f23c11d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:13.674 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:13.671+0000 7f23c11d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:14.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:42:14.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:14.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:14.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:14.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:14.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:14.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:14.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:14.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:14.011 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:42:14.189 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:14.187+0000 7f23c11d4ec0 -1 Falling back to public interface 2020-03-11T22:42:14.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:14.967 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:14.964+0000 7f23c11d4ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:42:15.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:15.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:15.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:15.430 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:42:15.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:15.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:15.874 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43924,v1:127.0.0.1:6811/43924] [v2:127.0.0.1:6812/43924,v1:127.0.0.1:6813/43924] exists,up 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:42:15.874 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:42:15.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:15.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:15.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:15.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:42:15.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:42:15.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:15.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:15.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:15.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:42:15.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:15.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:15.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:15.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:42:15.880 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:15.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:42:15.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5' 2020-03-11T22:42:15.881 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:15.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBHaWle/hILNRAAsK8sngEfzyjGbEOGZxsIeA== 2020-03-11T22:42:15.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBHaWle/hILNRAAsK8sngEfzyjGbEOGZxsIeA=="}' 2020-03-11T22:42:15.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 -i td/divergent-priors/2/new.json 2020-03-11T22:42:16.355 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:42:16.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:42:16.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHaWle/hILNRAAsK8sngEfzyjGbEOGZxsIeA== --osd-uuid 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:42:16.396 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.393+0000 7f92d5007ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.417 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.415+0000 7f92d5007ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.419 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.416+0000 7f92d5007ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.419 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.417+0000 7f92d5007ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:16.419 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.417+0000 7f92d5007ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:16.420 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.417+0000 7f92d5007ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:16.476 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:16.473+0000 7f92d5007ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:42:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:42:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:19.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:42:19.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:19.796 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:42:20.283 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:42:20.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:42:20.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:20.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:20.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:20.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:20.311 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:20.309+0000 7ff2d4729ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.313 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:20.311+0000 7ff2d4729ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.315 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:20.312+0000 7ff2d4729ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:20.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:20.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:20.652 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:42:20.829 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:20.827+0000 7ff2d4729ec0 -1 Falling back to public interface 2020-03-11T22:42:21.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:21.617 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:42:21.613+0000 7ff2d4729ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:42:22.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:22.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:22.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:22.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:22.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:22.051 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:42:22.423 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:23.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:23.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:23.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:42:23.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:23.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:23.425 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:42:23.800 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44279,v1:127.0.0.1:6819/44279] [v2:127.0.0.1:6820/44279,v1:127.0.0.1:6821/44279] exists,up 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:42:23.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:23.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:23.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:23.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:42:24.536 INFO:tasks.workunit.client.0.smithi045.stderr:noout is set 2020-03-11T22:42:24.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:42:25.271 INFO:tasks.workunit.client.0.smithi045.stderr:noin is set 2020-03-11T22:42:25.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:42:25.989 INFO:tasks.workunit.client.0.smithi045.stderr:nodown is set 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:42:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:42:26.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:42:26.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:42:27.237 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:42:27.245 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:42:28.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:42:28.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:42:29.036 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:42:29.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:42:29.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:29.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:29.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:29.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:29.046 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:29.047 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:29.047 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:29.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:29.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:29.047 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:29.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:29.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:29.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:29.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:29.143 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:29.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:29.514 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:42:29.514 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:42:29.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:29.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:29.515 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:29.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:42:29.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:42:29.651 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:29.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:29.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:42:29.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:42:29.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:29.805 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:29.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:29.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:42:29.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:42:29.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:29.928 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:42:30.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:42:30.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:31.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:31.292 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:31.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836486 2020-03-11T22:42:31.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:31.672 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:42:31.672 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:31.673 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:31.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:42:31.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:31.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:42:31.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:42:31.675 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:31.675 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:42:32.036 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:42:32.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:32.037 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:42:32.037 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:32.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:32.038 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:42:32.039 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:32.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:42:32.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:42:32.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:32.040 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:42:32.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:42:32.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:32.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:32.406 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:32.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:32.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:32.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:33.231 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:33.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:33.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:42:33.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:33.705 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:34.085 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:34.086 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:42:34.086 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:42:34.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:34.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:34.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:34.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:42:34.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:42:34.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:34.222 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:34.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:42:34.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2020-03-11T22:42:34.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:34.354 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:34.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:42:34.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509447' 2020-03-11T22:42:34.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:34.489 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:34.490 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:34.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:34.491 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:34.492 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:34.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:42:34.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:42:34.492 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:34.493 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:42:34.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:42:34.899 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:34.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:42:34.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:34.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:34.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:42:34.902 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:34.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:42:34.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:42:34.903 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:34.903 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:42:35.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:42:35.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:35.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:35.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:35.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:35.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:35.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:35.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:42:35.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:42:35.267 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:42:35.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509445 -lt 64424509447 2020-03-11T22:42:35.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:36.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:36.635 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:37.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509447 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:37.008 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:37.009 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:37.009 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:37.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:37.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:37.010 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:37.105 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:37.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:37.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:37.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:37.106 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.468 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:37.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:42:37.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:42:37.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:37.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:42:37.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972' 2020-03-11T22:42:37.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.741 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:37.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:42:37.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509449' 2020-03-11T22:42:37.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:37.877 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:42:37.877 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:37.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:37.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:42:37.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:37.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:42:37.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:42:37.879 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:42:37.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:38.241 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836492 2020-03-11T22:42:38.241 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:39.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:39.243 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:39.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:42:39.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:39.634 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:39.634 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:39.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:39.635 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:39.636 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:39.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:42:39.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:42:39.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:39.637 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:42:40.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:42:40.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:40.002 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:42:40.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:40.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:40.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:42:40.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:42:40.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509449 2020-03-11T22:42:40.390 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:40.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:40.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:40.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:40.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:40.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:41.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:41.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:41.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:41.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:41.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:41.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:41.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:41.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:42:41.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi045.stdout:primary and soon to be divergent is 1 2020-03-11T22:42:42.392 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:42.392 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:42:28.412844+0000 0'0 28:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:42:42.392 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:42:42.392 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:42.393 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:42:42.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:42:42.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:42.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:42:42.403 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:42:42.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:42:42.404 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:42:42.404 INFO:tasks.workunit.client.0.smithi045.stdout:writing initial objects 2020-03-11T22:42:42.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:42:42.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:42:42.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:42:42.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:42:42.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.727 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:42:42.828 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.828 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:42:42.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:42:42.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:42:42.945 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.945 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:42:42.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:42.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:42:43.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:42:43.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:42:43.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:42:43.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:42:43.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:42:43.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:42:43.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:42:43.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:42:43.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:42:43.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:42:43.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:42:43.819 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:42:43.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:42:43.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:43.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:42:44.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:42:44.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:42:44.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.129 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:42:44.169 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.169 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:42:44.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:42:44.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:42:44.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:42:44.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:42:44.487 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:42:44.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:42:44.621 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.621 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:42:44.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:42:44.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:42:44.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:42:44.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:44.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:42:45.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:42:45.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:42:45.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:42:45.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:42:45.337 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:42:45.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:42:45.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:42:45.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:42:45.553 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.553 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:42:45.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:42:45.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:42:45.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:42:45.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:42:45.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:42:45.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:45.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:42:46.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:42:46.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:42:46.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:42:46.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:42:46.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:42:46.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:42:46.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:42:46.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:42:46.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:42:46.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:42:46.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:42:46.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:42:46.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:42:46.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:42:46.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:46.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:42:47.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:42:47.066 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.066 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:42:47.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:42:47.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:42:47.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:42:47.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:42:47.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:42:47.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:42:47.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:42:47.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:42:47.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:42:47.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:42:47.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:42:47.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.812 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:42:47.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:42:47.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:47.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:42:48.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:42:48.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:42:48.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:42:48.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:42:48.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:42:48.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:42:48.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:42:48.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:42:48.608 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.608 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:42:48.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:42:48.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:42:48.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:42:48.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:42:48.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:48.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:48.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:48.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:48.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:48.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:48.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:48.949 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:48.950 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:48.950 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:48.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:48.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:48.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:49.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:49.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:49.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:49.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:49.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:49.374 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:49.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:42:49.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:42:49.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:49.511 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672976 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976' 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509454 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976 2-64424509454' 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:42:50.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:42:50.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:51.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:51.210 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:51.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836496 2020-03-11T22:42:51.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:51.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672976 2020-03-11T22:42:51.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:51.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:51.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672976 2020-03-11T22:42:51.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:51.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672976 2020-03-11T22:42:51.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2020-03-11T22:42:51.588 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:51.589 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672976 2020-03-11T22:42:51.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 42949672976 2020-03-11T22:42:51.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:51.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509454 2020-03-11T22:42:51.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:51.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:51.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509454 2020-03-11T22:42:51.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:51.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509454 2020-03-11T22:42:51.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2020-03-11T22:42:51.977 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509454 2020-03-11T22:42:51.978 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:52.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509454 -lt 64424509454 2020-03-11T22:42:52.346 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:52.346 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:52.346 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:52.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:52.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:52.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:52.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:52.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:52.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:52.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:52.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:53.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:53.214 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:53.214 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:53.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:53.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:53.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:53.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:53.682 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:42:53.682 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:42:53.682 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:42:53.682 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:42:53.682 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:42:54.048 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:42:54.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:42:54.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:54.049 INFO:tasks.workunit.client.0.smithi045.stdout:blackholing osds 0 2 2020-03-11T22:42:54.429 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:54.430 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:42:28.412844+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:42:54.430 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:42:54.430 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:54.431 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:42:54.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:54.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:42:54.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:42:54.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:42:54.440 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:54.441 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:54.441 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:54.441 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:42:54.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:42:54.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:42:54.532 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:42:54.532 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:42:54.532 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:42:54.539 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:54.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:42:54.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:42:54.540 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:42:54.540 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:54.540 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:54.541 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:42:54.541 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:42:54.541 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:42:54.541 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:42:54.632 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:42:54.632 INFO:tasks.workunit.client.0.smithi045.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:42:54.632 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:42:54.639 INFO:tasks.workunit.client.0.smithi045.stdout:writing divergent object 2020-03-11T22:42:54.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:42:54.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:42:54.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:42:55.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:42:55.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:55.795 INFO:tasks.workunit.client.0.smithi045.stdout:created pool test snap snap1 2020-03-11T22:42:55.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:42:55.797 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:42:55.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:43:05.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:43:05.803 INFO:tasks.workunit.client.0.smithi045.stderr:rados: no process found 2020-03-11T22:43:05.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:43:05.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:05.804 INFO:tasks.workunit.client.0.smithi045.stdout:killing all the osds 2020-03-11T22:43:06.153 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:06.153 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:42:28.412844+0000 30'104 30:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:43:06.153 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:43:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:06.155 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:43:06.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:43:06.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:06.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:06.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:06.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:06.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:06.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:06.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:06.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:43:06.900 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already down. 2020-03-11T22:43:06.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:06.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:43:07.429 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already down. 2020-03-11T22:43:07.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:07.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:43:07.982 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already down. 2020-03-11T22:43:07.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:07.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:43:08.543 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already out. 2020-03-11T22:43:08.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:08.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:43:09.075 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already out. 2020-03-11T22:43:09.084 INFO:tasks.workunit.client.0.smithi045.stdout:bringing up non_divergent 0 2 2020-03-11T22:43:09.084 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:43:09.084 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:09.424 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:09.424 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:42:28.412844+0000 30'104 30:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:43:09.424 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:43:09.424 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:09.426 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:09.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:09.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:09.439 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:09.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:09.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:09.442 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:43:09.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:09.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:09.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:09.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:09.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:09.462+0000 7f87eca9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:09.466 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:09.463+0000 7f87eca9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:09.465+0000 7f87eca9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:09.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:09.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:09.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:09.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:09.804 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:43:09.984 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:09.981+0000 7f87eca9eec0 -1 Falling back to public interface 2020-03-11T22:43:10.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:10.846 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:10.844+0000 7f87eca9eec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:43:11.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:11.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:11.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:11.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:11.167 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:43:11.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:11.560 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/48710,v1:127.0.0.1:6803/48710] [v2:127.0.0.1:6804/48710,v1:127.0.0.1:6805/48710] exists,up ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:43:11.560 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:43:11.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:11.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:11.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:11.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:11.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:43:11.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:11.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:11.563 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:11.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:11.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:11.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:11.566 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:11.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:11.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:11.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:11.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:11.587 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:11.584+0000 7ff22333bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:11.588 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:11.586+0000 7ff22333bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:11.590 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:11.587+0000 7ff22333bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:11.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:11.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:11.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:11.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:11.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:11.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:11.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:11.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:11.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:11.925 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:43:12.105 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:12.102+0000 7ff22333bec0 -1 Falling back to public interface 2020-03-11T22:43:12.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:12.988 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:12.985+0000 7ff22333bec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:43:13.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:13.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:13.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:13.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:13.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:13.333 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:43:13.712 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/48904,v1:127.0.0.1:6811/48904] [v2:127.0.0.1:6812/48904,v1:127.0.0.1:6813/48904] exists,up 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:43:13.712 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:13.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:13.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:13.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:13.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:43:14.221 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already in. 2020-03-11T22:43:14.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:14.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:43:14.725 INFO:tasks.workunit.client.0.smithi045.stderr:osd.2 is already in. 2020-03-11T22:43:14.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:43:19.735 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:43:19.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:43:19.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:43:19.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:19.736 INFO:tasks.workunit.client.0.smithi045.stdout:writing non-divergent object existing_4 2020-03-11T22:43:20.090 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:20.090 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:15.016036+0000 29'100 40:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:43:20.090 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:43:20.090 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:20.091 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:43:20.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:43:20.148 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:20.148 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:43:20.148 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:20.148 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:20.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:20.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:20.149 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:20.150 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:20.150 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:20.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:20.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:20.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:20.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:20.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:20.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:20.209 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:20.210 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:20.592 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822660 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822660' 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:20.824 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:20.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:20.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822660 1-' 2020-03-11T22:43:20.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:20.832 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:20.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789956 2020-03-11T22:43:20.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822660 1- 2-158913789956' 2020-03-11T22:43:20.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:20.971 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822660 2020-03-11T22:43:20.971 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:20.972 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:20.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822660 2020-03-11T22:43:20.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:20.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822660 2020-03-11T22:43:20.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2020-03-11T22:43:20.973 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:20.973 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 154618822660 2020-03-11T22:43:21.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822658 -lt 154618822660 2020-03-11T22:43:21.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:22.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:22.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:22.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822661 -lt 154618822660 2020-03-11T22:43:22.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:22.713 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:22.713 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:22.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:22.714 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:22.714 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:22.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:22.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:22.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:22.715 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:43:23.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:43:23.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:43:23.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:23.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789956 2020-03-11T22:43:23.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:23.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:23.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789956 2020-03-11T22:43:23.081 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:23.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789956 2020-03-11T22:43:23.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2020-03-11T22:43:23.082 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 158913789956 2020-03-11T22:43:23.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:23.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789956 -lt 158913789956 2020-03-11T22:43:23.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:23.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:23.447 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:23.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:23.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:23.930 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:23.930 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:23.930 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:23.931 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:23.931 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:23.931 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:24.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:24.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:24.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:24.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:24.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:24.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:24.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:24.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:24.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:24.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:24.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:24.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:24.870 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:24.870 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:24.870 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:24.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:24.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:24.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:25.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:25.222 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:25.222 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:25.222 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:25.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:25.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:25.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:25.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:25.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:25.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:25.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:25.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:25.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:26.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:26.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:26.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:43:26.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:26.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:43:26.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:26.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:26.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:26.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:26.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:26.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:26.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:26.373 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:26.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:26.729 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:26.729 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:26.730 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:27.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:27.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:27.247 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:27.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:27.721 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:27.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:43:27.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:27.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:43:28.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:28.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:28.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:28.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:28.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:28.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:28.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:28.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:28.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:28.483 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:28.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:28.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:28.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:28.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:28.960 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:29.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:29.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:29.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:43:29.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:29.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:43:30.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:30.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:30.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:30.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:30.233 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:30.233 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:30.233 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:30.233 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:30.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:30.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:30.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:30.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:31.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:31.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:31.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:31.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:31.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:31.543 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:31.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:43:31.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:31.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:43:33.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:33.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:33.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:33.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:33.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:33.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:33.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:33.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:33.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:33.506 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:33.506 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:33.506 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:33.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:33.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:33.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:33.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:33.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:33.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:33.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:33.977 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:33.977 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:34.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:34.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:34.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:43:34.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:34.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:43:37.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:37.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:37.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:37.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:37.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:37.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:37.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:37.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:38.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:38.012 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:38.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:38.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:38.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:38.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:38.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:38.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:38.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:38.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:38.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:43:38.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:38.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:43:45.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:45.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:45.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:45.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:45.729 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:45.729 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:45.729 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:46.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:46.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:46.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:46.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:46.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:46.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:46.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:43:46.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:46.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:43:53.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:53.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:53.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:53.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:53.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:53.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:53.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:53.998 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:54.362 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:54.363 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:54.363 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:54.363 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:54.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:54.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:54.833 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:55.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:55.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:55.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:43:55.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:43:55.773 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:43:55.773 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "b153d209-e92b-4d28-bd13-ebc613adc942", 2020-03-11T22:43:55.773 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.773 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:43:55.773 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:43:55.769561+0000", 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.774 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:43:55.775 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.776 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.777 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:43:55.778 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.779 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.779 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:43:55.779 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.779 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:43:55.779 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.780 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:43:55.780 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:43:55.780 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:43:55.780 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:43:55.781 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:42:01.432470Z", 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:42:01.432470Z", 2020-03-11T22:43:55.783 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:43:55.784 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.785 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:43:55.786 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:43:55.787 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.787 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.787 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.787 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:43:55.787 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:43:55.788 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:43:55.789 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:43:55.789 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.789 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.789 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.790 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:43:55.790 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:43:55.790 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.791 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:43:55.791 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 40, 2020-03-11T22:43:55.791 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:43:55.791 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:42:02.181862+0000", 2020-03-11T22:43:55.791 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:43:14.761688+0000", 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:43:13.059532+0000", 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:43:14.553135+0000", 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:43:55.792 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:43:55.793 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 8, 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:43:55.794 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.795 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:42:26.941184+0000", 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 16389, 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 3, 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:43:55.796 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 1, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 1, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 1, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 1, 2020-03-11T22:43:55.797 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 1, 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:43:55.798 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "30", 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:43:55.799 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "pool", 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 1, 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 30, 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [ 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "snapid": 1, 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "stamp": "2020-03-11T22:42:55.671198+0000", 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "snap1" 2020-03-11T22:43:55.800 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:43:55.801 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:43:55.802 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.803 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:43:55.804 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 8192, 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.805 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.806 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:43:55.806 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.806 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:43:55.806 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "ced68667-c2e2-48d3-8915-2f9a40be4ea1", 2020-03-11T22:43:55.806 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:43:55.807 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:43:55.807 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:43:55.807 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.807 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 5, 2020-03-11T22:43:55.807 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 36, 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 39, 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 31, 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.808 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:43:55.809 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.810 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:43:55.811 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.812 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.813 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.814 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48710 2020-03-11T22:43:55.815 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/48710", 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/48710", 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48710", 2020-03-11T22:43:55.816 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48710", 2020-03-11T22:43:55.817 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:43:55.817 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:43:55.817 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:43:55.817 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.817 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "5b96c0dd-0926-4a5f-acff-35ff3d1763e2", 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:43:55.818 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 25, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:43:55.819 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.820 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.821 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:55.822 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.823 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.824 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.825 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.826 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:43:55.827 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/43924", 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/43924", 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43924", 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43924", 2020-03-11T22:43:55.828 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5", 2020-03-11T22:43:55.829 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 15, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 29, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 37, 2020-03-11T22:43:55.830 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 33, 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:55.831 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.832 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.832 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.832 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.833 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.834 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.835 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.835 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.835 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:55.835 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.835 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.837 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:55.838 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 48904 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.839 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/48904", 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/48904", 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48904", 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48904", 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:43:55.840 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:43:55.841 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:43:06.675738+0000", 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:09.426123+0000", 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:43:55.842 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:43:07.234896+0000", 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:43:55.843 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:15.950065+0000", 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:43:07.748718+0000", 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:22.662198+0000", 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.845 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:43:55.846 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "myprofile": { 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-device-class": "", 2020-03-11T22:43:55.847 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-failure-domain": "osd", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-root": "default", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "1", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van", 2020-03-11T22:43:55.848 INFO:tasks.workunit.client.0.smithi045.stdout: "w": "8" 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:43:55.849 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/48710,v1:127.0.0.1:6805/48710]", 2020-03-11T22:43:55.850 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.851 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:43:55.853 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/48710,v1:127.0.0.1:6803/48710]", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48710,v1:127.0.0.1:6809/48710]", 2020-03-11T22:43:55.854 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48710,v1:127.0.0.1:6807/48710]", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:43:55.855 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.856 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.856 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:43:55.856 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:43:55.856 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.856 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/43924,v1:127.0.0.1:6813/43924]", 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:43:55.857 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.858 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.859 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.860 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/43924,v1:127.0.0.1:6811/43924]", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43924,v1:127.0.0.1:6817/43924]", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43924,v1:127.0.0.1:6815/43924]", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:43:55.862 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/48904,v1:127.0.0.1:6813/48904]", 2020-03-11T22:43:55.863 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.864 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.865 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.866 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/48904,v1:127.0.0.1:6811/48904]", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48904,v1:127.0.0.1:6817/48904]", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48904,v1:127.0.0.1:6815/48904]", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 40, 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:43:55.870 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.871 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.872 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:43:55.872 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:43:55.872 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:43:55.872 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.873 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.873 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:43:55.873 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.873 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:43:55.873 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:43:55.874 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.875 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.876 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.877 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:43:55.878 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:43:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:43:55.880 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:43:55.881 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.882 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:43:55.883 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.884 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.885 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:43:55.886 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:43:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:43:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.888 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:43:55.889 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:43:55.890 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.891 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.891 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:43:55.891 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.891 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:43:55.891 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.892 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:43:55.893 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:43:55.894 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.895 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.895 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:43:55.895 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:43:55.895 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:43:55.895 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 1, 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "test", 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 1, 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 3, 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 3, 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 3, 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:43:55.897 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.898 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:43:55.898 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 5 2020-03-11T22:43:55.898 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.898 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.899 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "set_choose_tries", 2020-03-11T22:43:55.899 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 100 2020-03-11T22:43:55.899 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.899 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_indep", 2020-03-11T22:43:55.900 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.901 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:43:55.902 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:43:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 1, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:43:55.904 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.905 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:43:55.906 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:43:55.906 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:43:55.907 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.908 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:43:55.909 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:43:55.910 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 1, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48430, 2020-03-11T22:43:55.911 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 101, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 1, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 303, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:55.912 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:43:55.913 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:43:55.914 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:43:55.915 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.917 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:43:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 18, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:55.919 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 3, 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 3, 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:43:55.920 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:43:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:43:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:43:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179904, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33856, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 11, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145716, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392896, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318866325504, 2020-03-11T22:43:55.922 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 34668544, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1523781, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 11440, 2020-03-11T22:43:55.923 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221214032 2020-03-11T22:43:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:43:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:43:55.926 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.927 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:55.928 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3179904, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 33856, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 11, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145716, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311392896, 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:43:55.929 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318866325504, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 34668544, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1523781, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.930 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 11440, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221214032 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:55.931 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:43:55.933 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48430, 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 101, 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 1, 2020-03-11T22:43:55.934 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 303, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:43:55.935 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:55.936 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:55.937 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:43:55.938 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:43:55.938 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:43:55.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:43:55.940 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:55.941 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:43:55.942 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:43:55.943 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:43:55.943 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:43:55.943 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19857408, 2020-03-11T22:43:55.943 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1241088, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.944 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 18, 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 3, 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 3, 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.945 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 154618822667 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:43:55.946 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672979 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 158913789963 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:43:55.947 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 1 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.949 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2147483647, 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.950 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:43:55.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:43:55.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:43:55.951 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.951 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.951 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 1, 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:43:55.952 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:43:55.953 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:42:02.182192+0000", 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 338, 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:43:55.954 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:43:55.955 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:43:55.956 INFO:tasks.workunit.client.0.smithi045.stderr:report 1216682157 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:55.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:55.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:43:55.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:55.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:43:55.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:43:56.410 INFO:tasks.workunit.client.0.smithi045.stderr:PG '1.0' not found 2020-03-11T22:43:56.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:56.931 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:56.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:56.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:56.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:56.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:56.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:56.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:56.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:56.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:56.933 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:56.933 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:56.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:56.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:56.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:56.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:56.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:56.938 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:43:56.962 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:56.960+0000 7fd94c9a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:56.964 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:56.962+0000 7fd94c9a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:56.966 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:56.964+0000 7fd94c9a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:57.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:57.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:57.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:57.312 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:57.313 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:43:57.483 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:57.481+0000 7fd94c9a3ec0 -1 Falling back to public interface 2020-03-11T22:43:57.699 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 36 up_thru 39 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/48710,v1:127.0.0.1:6803/48710] [v2:127.0.0.1:6804/48710,v1:127.0.0.1:6805/48710] exists,up ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:43:57.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:57.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:57.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:43:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:57.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:43:57.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:57.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:43:57.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:43:58.443 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:58.440+0000 7fd94c9a3ec0 -1 osd.0 40 log_to_monitors {default=true} 2020-03-11T22:43:58.616 INFO:tasks.workunit.client.0.smithi045.stderr:PG '1.0' not found 2020-03-11T22:43:59.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:43:59.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:43:59.134 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:59.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:59.136 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:59.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:59.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:59.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:59.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:59.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:59.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:59.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:59.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:59.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:59.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:59.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:59.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:59.141 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:43:59.166 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:59.163+0000 7f0ce785cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:59.168 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:59.166+0000 7f0ce785cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:59.170 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:59.167+0000 7f0ce785cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:59.512 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:59.512 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:59.513 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:59.514 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:43:59.685 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:43:59.683+0000 7f0ce785cec0 -1 Falling back to public interface 2020-03-11T22:43:59.908 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/48904,v1:127.0.0.1:6811/48904] [v2:127.0.0.1:6812/48904,v1:127.0.0.1:6813/48904] exists,up 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:43:59.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:59.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:59.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:59.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:43:59.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:59.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:59.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:59.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:59.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:59.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:44:00.701 INFO:tasks.workunit.client.0.smithi045.stderr:PG '1.0' not found 2020-03-11T22:44:00.735 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:00.732+0000 7f0ce785cec0 -1 osd.2 40 log_to_monitors {default=true} 2020-03-11T22:44:01.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:01.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:01.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:01.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:01.219 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:01.219 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:01.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:01.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:01.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:01.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:01.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:01.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:01.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:01.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:01.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:01.761 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:44:01.761 INFO:tasks.workunit.client.0.smithi045.stderr:2' 2020-03-11T22:44:01.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:01.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:01.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:01.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=180388626435 2020-03-11T22:44:01.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435' 2020-03-11T22:44:01.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:01.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:01.994 INFO:tasks.workunit.client.0.smithi045.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:02.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:02.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1-' 2020-03-11T22:44:02.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:02.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:02.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=193273528323 2020-03-11T22:44:02.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1- 2-193273528323' 2020-03-11T22:44:02.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:02.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:02.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:02.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:02.141 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:02.141 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:02.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=180388626435 2020-03-11T22:44:02.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 180388626435' 2020-03-11T22:44:02.142 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 180388626435 2020-03-11T22:44:02.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:02.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 180388626435 2020-03-11T22:44:02.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:03.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:03.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:03.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 180388626435 -lt 180388626435 2020-03-11T22:44:03.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:03.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:03.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:03.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:03.884 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:03.884 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:03.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:03.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:03.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:03.885 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=193273528323 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 193273528323 2020-03-11T22:44:04.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 193273528323 -lt 193273528323 2020-03-11T22:44:04.687 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:04.687 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:04.687 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:05.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:05.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:05.217 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:05.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:05.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:05.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:05.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:05.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:05.572 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:05.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:05.573 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:05.573 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:06.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:06.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:06.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:06.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:06.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:06.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:06.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:06.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:06.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:06.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:06.499 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:06.499 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:06.499 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:06.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:06.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:06.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:06.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:06.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:06.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:06.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:06.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:06.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:07.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:07.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:07.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:07.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:07.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:07.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:07.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:07.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:07.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:07.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:07.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:07.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:07.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:08.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:08.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:08.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:08.004 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:08.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:08.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:08.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:08.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:08.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:08.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:08.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:08.475 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:08.475 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:08.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:08.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:08.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:08.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:08.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:09.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:09.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:09.351 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:09.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:09.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:09.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:09.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:10.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:10.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:10.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:10.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:10.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:10.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:10.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:10.186 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:10.186 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:10.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:10.703 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:10.703 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:10.703 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:10.703 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:11.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:11.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:11.505 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:11.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:11.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:11.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:11.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:12.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:12.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:12.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:12.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:12.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:14.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:14.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:14.440 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:14.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:14.801 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:14.801 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:14.801 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:15.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:15.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:15.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:15.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:15.756 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:15.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:15.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:15.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:15.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:18.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:18.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:18.958 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:18.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:18.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:18.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:18.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:18.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:19.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:19.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:19.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:19.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:19.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:19.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:19.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:19.794 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:20.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:20.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:20.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:44:20.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:20.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:26.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:26.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:26.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:27.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:27.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:27.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:27.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:27.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:27.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:27.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:27.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:27.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:27.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:27.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:27.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:27.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:35.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:35.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:35.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:35.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:35.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:35.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:35.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:35.278 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:35.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:35.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:35.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:35.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:36.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:36.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:36.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:36.112 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:36.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:36.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:36.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:36.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:37.078 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:44:37.078 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_fingerprint": "b153d209-e92b-4d28-bd13-ebc613adc942", 2020-03-11T22:44:37.078 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:37.078 INFO:tasks.workunit.client.0.smithi045.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:37.078 INFO:tasks.workunit.client.0.smithi045.stdout: "timestamp": "2020-03-11T22:44:37.074671+0000", 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "tag": "", 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "health": { 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "checks": { 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:44:37.079 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 17 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [], 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: "OSD_DOWN": { 2020-03-11T22:44:37.080 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 osds down", 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.081 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "osd.1 (root=default,host=smithi045) is down" 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: "PG_DEGRADED": { 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:37.082 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:44:37.083 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:44:37.083 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:44:37.083 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.083 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:44:37.083 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.084 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:44:37.084 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.084 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "summary": { 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: "count": 1 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: "detail": [ 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:37.086 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: "muted": false 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: "mutes": [] 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.087 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "monmap": { 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:42:01.432470Z", 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:42:01.432470Z", 2020-03-11T22:44:37.088 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release": 15, 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "features": { 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "persistent": [ 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "kraken", 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "luminous", 2020-03-11T22:44:37.089 INFO:tasks.workunit.client.0.smithi045.stdout: "mimic", 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap-prune", 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: "nautilus", 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: "octopus" 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: "optional": [] 2020-03-11T22:44:37.090 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "mons": [ 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "rank": 0, 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "a", 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.091 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 0 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:37.092 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 0, 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 0 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: "quorum": [ 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: 0 2020-03-11T22:44:37.093 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap": { 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 46, 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "fsid": "e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3", 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "created": "2020-03-11T22:42:02.181862+0000", 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:44:01.276359+0000", 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "last_up_change": "2020-03-11T22:44:01.021198+0000", 2020-03-11T22:44:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: "last_in_change": "2020-03-11T22:43:14.553135+0000", 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_num": 5800384, 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_set": [ 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "nodown", 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "noin", 2020-03-11T22:44:37.095 INFO:tasks.workunit.client.0.smithi045.stdout: "noout", 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "pglog_hardlimit", 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snapdirs", 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "recovery_deletes", 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "sortbitwise" 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_version": 8, 2020-03-11T22:44:37.096 INFO:tasks.workunit.client.0.smithi045.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_snapshot": "", 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_max": 1, 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "max_osd": 3, 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:37.097 INFO:tasks.workunit.client.0.smithi045.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "pools": [ 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_name": "test", 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "create_time": "2020-03-11T22:42:26.941184+0000", 2020-03-11T22:44:37.098 INFO:tasks.workunit.client.0.smithi045.stdout: "flags": 16389, 2020-03-11T22:44:37.099 INFO:tasks.workunit.client.0.smithi045.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:44:37.099 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 3, 2020-03-11T22:44:37.099 INFO:tasks.workunit.client.0.smithi045.stdout: "size": 3, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 2, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_rule": 1, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "object_hash": 2, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num": 1, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num": 1, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_placement_num_target": 1, 2020-03-11T22:44:37.100 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_target": 1, 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_num_pending": 1, 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "source_pgid": "0.0", 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "ready_epoch": 0, 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_started": 0, 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:37.101 INFO:tasks.workunit.client.0.smithi045.stdout: "source_version": "0'0", 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "target_version": "0'0" 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "last_change": "30", 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:44:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: "auid": 0, 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_mode": "pool", 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_seq": 1, 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_epoch": 30, 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_snaps": [ 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.103 INFO:tasks.workunit.client.0.smithi045.stdout: "snapid": 1, 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "stamp": "2020-03-11T22:42:55.671198+0000", 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "snap1" 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps": "[]", 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "quota_max_objects": 0, 2020-03-11T22:44:37.104 INFO:tasks.workunit.client.0.smithi045.stdout: "tiers": [], 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "tier_of": -1, 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "read_tier": -1, 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "write_tier": -1, 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_mode": "none", 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_bytes": 0, 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "target_max_objects": 0, 2020-03-11T22:44:37.105 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_params": { 2020-03-11T22:44:37.106 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "none" 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_period": 0, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_count": 0, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:37.107 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "grade_table": [], 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "stripe_width": 8192, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "expected_num_objects": 0, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "fast_read": false, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "options": {}, 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: "application_metadata": {} 2020-03-11T22:44:37.108 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: "osds": [ 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "ced68667-c2e2-48d3-8915-2f9a40be4ea1", 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:44:37.109 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 36, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 40, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 42, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 45, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 41, 2020-03-11T22:44:37.110 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.112 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.113 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.114 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.115 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:37.116 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.117 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50353 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.118 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6803/50353", 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6805/50353", 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50353", 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50353", 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "5b96c0dd-0926-4a5f-acff-35ff3d1763e2", 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 0, 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:44:37.120 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 0, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 0, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 10, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 25, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 32, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:44:37.121 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.122 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:44:37.123 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.124 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.125 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.126 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:37.127 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.127 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.127 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.127 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.127 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:37.128 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.128 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.128 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.128 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 43924 2020-03-11T22:44:37.129 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.130 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.130 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.130 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/43924", 2020-03-11T22:44:37.130 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/43924", 2020-03-11T22:44:37.130 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43924", 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43924", 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: "exists" 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "uuid": "4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5", 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 1, 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "in": 1, 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 1, 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_affinity": 1, 2020-03-11T22:44:37.132 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_begin": 37, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "last_clean_end": 40, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 45, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "up_thru": 0, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "down_at": 44, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "lost_at": 0, 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addrs": { 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.133 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.134 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addrs": { 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.135 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.138 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: "addrvec": [ 2020-03-11T22:44:37.139 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v2", 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "v1", 2020-03-11T22:44:37.140 INFO:tasks.workunit.client.0.smithi045.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: "nonce": 50585 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: "public_addr": "127.0.0.1:6811/50585", 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: "cluster_addr": "127.0.0.1:6813/50585", 2020-03-11T22:44:37.141 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50585", 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50585", 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: "state": [ 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: "exists", 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: "up" 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.142 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_xinfo": [ 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:43:58.544661+0000", 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:44:37.143 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:09.426123+0000", 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:43:07.234896+0000", 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:44:37.145 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:15.950065+0000", 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: "down_stamp": "2020-03-11T22:44:00.816016+0000", 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_probability": 0, 2020-03-11T22:44:37.146 INFO:tasks.workunit.client.0.smithi045.stdout: "laggy_interval": 0, 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "features": 4540138292836696063, 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "old_weight": 0, 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:22.662198+0000", 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "dead_epoch": 0 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap": [], 2020-03-11T22:44:37.147 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_upmap_items": [], 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_temp": [], 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "primary_temp": [], 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "blacklist": {}, 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "erasure_code_profiles": { 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "default": { 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:44:37.148 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "2", 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "myprofile": { 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-device-class": "", 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-failure-domain": "osd", 2020-03-11T22:44:37.149 INFO:tasks.workunit.client.0.smithi045.stdout: "crush-root": "default", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "k": "2", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "m": "1", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "plugin": "jerasure", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "technique": "reed_sol_van", 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: "w": "8" 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.150 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "new_removed_snaps": [], 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "new_purged_snaps": [], 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "crush_node_flags": {}, 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "device_class_flags": {} 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.151 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_metadata": [ 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6804/50353,v1:127.0.0.1:6805/50353]", 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:44:37.152 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:37.154 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:37.155 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6802/50353,v1:127.0.0.1:6803/50353]", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50353,v1:127.0.0.1:6809/50353]", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50353,v1:127.0.0.1:6807/50353]", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:44:37.156 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:37.157 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:44:37.158 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/43924,v1:127.0.0.1:6813/43924]", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:37.159 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:37.160 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:37.161 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:37.161 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:44:37.161 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:37.161 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:37.162 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:37.162 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:44:37.162 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:44:37.162 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:37.162 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/43924,v1:127.0.0.1:6811/43924]", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43924,v1:127.0.0.1:6817/43924]", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43924,v1:127.0.0.1:6815/43924]", 2020-03-11T22:44:37.163 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:37.164 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.165 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "arch": "x86_64", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "back_addr": "[v2:127.0.0.1:6812/50585,v1:127.0.0.1:6813/50585]", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "back_iface": "", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs": "1", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:37.166 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:37.167 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_release": "octopus", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "default_device_class": "hdd", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HPNL", 2020-03-11T22:44:37.168 INFO:tasks.workunit.client.0.smithi045.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": "sda", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "distro": "centos", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "distro_version": "8", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "front_addr": "[v2:127.0.0.1:6810/50585,v1:127.0.0.1:6811/50585]", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "front_iface": "", 2020-03-11T22:44:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50585,v1:127.0.0.1:6817/50585]", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50585,v1:127.0.0.1:6815/50585]", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "hostname": "smithi045", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "journal_rotational": "1", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "mem_total_kb": "32662100", 2020-03-11T22:44:37.170 INFO:tasks.workunit.client.0.smithi045.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: "os": "Linux", 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: "rotational": "1" 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.171 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "osdmap_last_committed": 46, 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "crushmap": { 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "devices": [ 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.0", 2020-03-11T22:44:37.172 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.1", 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:44:37.173 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd.2", 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: "class": "hdd" 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.174 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: "types": [ 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 0, 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "osd" 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.175 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:44:37.176 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "host" 2020-03-11T22:44:37.176 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.176 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.176 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 2, 2020-03-11T22:44:37.176 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "chassis" 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 3, 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "rack" 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 4, 2020-03-11T22:44:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "row" 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 5, 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pdu" 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 6, 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "pod" 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 7, 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "room" 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.179 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 8, 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "datacenter" 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 9, 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "zone" 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.180 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 10, 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "region" 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "root" 2020-03-11T22:44:37.181 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: "buckets": [ 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -1, 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default", 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:44:37.182 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:44:37.183 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.184 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:44:37.184 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.184 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.184 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.184 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -2, 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "default~hdd", 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 11, 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "root", 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:44:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.186 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -3, 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045", 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:44:37.187 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:44:37.188 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.189 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:44:37.190 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.190 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:44:37.191 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.191 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.191 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.191 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.191 INFO:tasks.workunit.client.0.smithi045.stdout: "id": -4, 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "name": "smithi045~hdd", 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "type_id": 1, 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "type_name": "host", 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 19206, 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "alg": "straw2", 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "hash": "rjenkins1", 2020-03-11T22:44:37.192 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 0, 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 0 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 1, 2020-03-11T22:44:37.193 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 1 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: "id": 2, 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: "weight": 6402, 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: "pos": 2 2020-03-11T22:44:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: "rules": [ 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 0, 2020-03-11T22:44:37.195 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 0, 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 1, 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 1, 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 10, 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.196 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_firstn", 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:44:37.197 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.199 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_id": 1, 2020-03-11T22:44:37.199 INFO:tasks.workunit.client.0.smithi045.stdout: "rule_name": "test", 2020-03-11T22:44:37.199 INFO:tasks.workunit.client.0.smithi045.stdout: "ruleset": 1, 2020-03-11T22:44:37.199 INFO:tasks.workunit.client.0.smithi045.stdout: "type": 3, 2020-03-11T22:44:37.199 INFO:tasks.workunit.client.0.smithi045.stdout: "min_size": 3, 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: "max_size": 3, 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: "steps": [ 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 5 2020-03-11T22:44:37.200 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "set_choose_tries", 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 100 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "take", 2020-03-11T22:44:37.201 INFO:tasks.workunit.client.0.smithi045.stdout: "item": -1, 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: "item_name": "default" 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "choose_indep", 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 0, 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: "type": "osd" 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "emit" 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: "tunables": { 2020-03-11T22:44:37.203 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_tries": 0, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_total_tries": 50, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:37.204 INFO:tasks.workunit.client.0.smithi045.stdout: "straw_calc_version": 1, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "profile": "jewel", 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "optimal_tunables": 1, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "legacy_tunables": 0, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:37.205 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v2_rules": 1, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v3_rules": 0, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "has_v5_rules": 0 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.206 INFO:tasks.workunit.client.0.smithi045.stdout: "choose_args": {} 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "fsmap": { 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "default_fscid": -1, 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": { 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "compat": {}, 2020-03-11T22:44:37.207 INFO:tasks.workunit.client.0.smithi045.stdout: "ro_compat": {}, 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "incompat": { 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:37.208 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: "feature_flags": { 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: "enable_multiple": false, 2020-03-11T22:44:37.209 INFO:tasks.workunit.client.0.smithi045.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: "standbys": [], 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: "filesystems": [] 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:37.210 INFO:tasks.workunit.client.0.smithi045.stdout: "auth": { 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 5, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "num_secrets": 3 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_active": 1, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osd": 3, 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_sum": { 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48430, 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 101, 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 1, 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 303, 2020-03-11T22:44:37.212 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:44:37.213 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:37.214 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:44:37.215 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:37.216 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:44:37.217 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 0, 2020-03-11T22:44:37.218 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 0, 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:44:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.220 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 18, 2020-03-11T22:44:37.220 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 3, 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 3, 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 0 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum": { 2020-03-11T22:44:37.221 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3181440, 2020-03-11T22:44:37.222 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 35392, 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 9, 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311391360, 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:44:37.223 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318864752640, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 36241408, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1562189, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 9484, 2020-03-11T22:44:37.224 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221215988 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:44:37.225 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:37.226 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_sum_by_class": { 2020-03-11T22:44:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: "hdd": { 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "up_from": 0, 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 0, 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pgs": 3, 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "num_osds": 3, 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:37.228 INFO:tasks.workunit.client.0.smithi045.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb": 314572800, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used": 3181440, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_data": 35392, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_omap": 9, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "kb_avail": 311391360, 2020-03-11T22:44:37.229 INFO:tasks.workunit.client.0.smithi045.stdout: "statfs": { 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 322122547200, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 318864752640, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 36241408, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1562189, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:44:37.230 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 9484, 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 3221215988 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "hb_peers": [], 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:37.231 INFO:tasks.workunit.client.0.smithi045.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: "op_queue_age_hist": { 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: "histogram": [], 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: "upper_bound": 1 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.232 INFO:tasks.workunit.client.0.smithi045.stdout: "perf_stat": { 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "apply_latency_ns": 0 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "alerts": [], 2020-03-11T22:44:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: "network_ping_times": [] 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: "pool_stats": [ 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: "poolid": 1, 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg": 1, 2020-03-11T22:44:37.234 INFO:tasks.workunit.client.0.smithi045.stdout: "stat_sum": { 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes": 48430, 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects": 101, 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_clones": 1, 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_object_copies": 303, 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_missing": 0, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_whiteouts": 0, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read": 0, 2020-03-11T22:44:37.236 INFO:tasks.workunit.client.0.smithi045.stdout: "num_read_kb": 0, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write": 101, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_write_kb": 101, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:37.237 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_omap": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_kb": 0, 2020-03-11T22:44:37.238 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_kb": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_promote": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:37.239 INFO:tasks.workunit.client.0.smithi045.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "num_omap_keys": 0, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "num_objects_repaired": 0 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.240 INFO:tasks.workunit.client.0.smithi045.stdout: "store_stats": { 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "total": 0, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "available": 0, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "internally_reserved": 0, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "allocated": 19857408, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "data_stored": 1241088, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed": 0, 2020-03-11T22:44:37.241 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "data_compressed_original": 0, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "omap_allocated": 0, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "internal_metadata": 0 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "log_size": 18, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:37.242 INFO:tasks.workunit.client.0.smithi045.stdout: "up": 3, 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: "acting": 3, 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: "num_store_stats": 3 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: "osd_stats": [ 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:44:37.243 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 180388626442 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 42949672979 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: "seq": 193273528330 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_state": [ 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:37.245 INFO:tasks.workunit.client.0.smithi045.stdout: "num": 1 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2, 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:44:37.246 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 2147483647, 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:44:37.247 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 1, 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 0, 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 0, 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.248 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: "osd": 0, 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: "num_primary_pg": 1, 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: "num_acting_pg": 1, 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.249 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": [ 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: "pool": 1, 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: "purged_snaps": {} 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: "servicemap": { 2020-03-11T22:44:37.250 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": 1, 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "modified": "2020-03-11T22:42:02.182192+0000", 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "services": {} 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "paxos": { 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "first_committed": 1, 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "last_committed": 438, 2020-03-11T22:44:37.251 INFO:tasks.workunit.client.0.smithi045.stdout: "last_pn": 0, 2020-03-11T22:44:37.252 INFO:tasks.workunit.client.0.smithi045.stdout: "accepted_pn": 0 2020-03-11T22:44:37.252 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:44:37.252 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:44:37.253 INFO:tasks.workunit.client.0.smithi045.stderr:report 108626604 2020-03-11T22:44:37.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:37.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:44:37.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:37.253 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery 2020-03-11T22:44:37.432 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:37.432 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:01.543515+0000 40'102 46:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:37.432 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:37.432 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:37.433 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:37.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:37.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.0.asok 2020-03-11T22:44:37.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:37.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:44:37.539 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:37.547 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:44:37.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.2.asok 2020-03-11T22:44:37.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:37.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:44:37.644 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:37.651 INFO:tasks.workunit.client.0.smithi045.stdout:revive divergent 1 2020-03-11T22:44:37.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:44:37.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:37.992 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:37.992 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:01.543515+0000 40'102 46:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:37.992 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:37.992 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:37.993 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:38.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:44:38.750 INFO:tasks.workunit.client.0.smithi045.stderr:noup is set 2020-03-11T22:44:38.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:38.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:38.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:38.763 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:38.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:38.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:38.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:38.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:38.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:38.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:38.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:38.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:38.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:38.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:38.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:38.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:38.767 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:44:38.789 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:38.785+0000 7fb8e4e0dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:38.791 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:38.787+0000 7fb8e4e0dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:38.793 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:38.789+0000 7fb8e4e0dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:44:39.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:44:39.308 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:39.305+0000 7fb8e4e0dec0 -1 Falling back to public interface 2020-03-11T22:44:40.157 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:40.144+0000 7fb8e4e0dec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:44:44.132 INFO:tasks.workunit.client.0.smithi045.stdout:delay recovery divergent 2020-03-11T22:44:44.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:44:44.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:44.488 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:44.488 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:01.543515+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:44.489 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:44.489 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:44.490 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:44.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:44:44.501 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:44:44.501 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:44:44.501 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:44.502 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:44.502 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:44:44.502 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17438/ceph-osd.1.asok 2020-03-11T22:44:44.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:44.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17438/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:44:44.596 INFO:tasks.workunit.client.0.smithi045.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:44.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:44:45.412 INFO:tasks.workunit.client.0.smithi045.stderr:noup is unset 2020-03-11T22:44:45.423 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:45.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:45.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:45.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:45.425 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:44:45.786 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 42 up_thru 45 down_at 41 last_clean_interval [36,40) [v2:127.0.0.1:6802/50353,v1:127.0.0.1:6803/50353] [v2:127.0.0.1:6804/50353,v1:127.0.0.1:6805/50353] exists,up ced68667-c2e2-48d3-8915-2f9a40be4ea1 2020-03-11T22:44:45.786 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:44:45.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:45.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:45.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:45.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:45.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:45.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:46.156 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/51973,v1:127.0.0.1:6819/51973] [v2:127.0.0.1:6820/51973,v1:127.0.0.1:6821/51973] exists,up 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:44:46.156 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:46.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:46.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:46.589 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 44 last_clean_interval [37,40) [v2:127.0.0.1:6810/50585,v1:127.0.0.1:6811/50585] [v2:127.0.0.1:6812/50585,v1:127.0.0.1:6813/50585] exists,up 4abe0b6a-5c9d-4475-b5b4-2e4e91d1ffb5 2020-03-11T22:44:46.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:46.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:46.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:46.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:46.929 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:46.929 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:01.543515+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:46.929 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:46.929 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:46.930 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:46.939 INFO:tasks.workunit.client.0.smithi045.stdout:wait for peering 2020-03-11T22:44:46.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:44:46.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:47.280 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:47.280 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:01.543515+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:47.280 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:47.280 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:47.281 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:47.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:44:57.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:44:57.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:57.565 INFO:tasks.workunit.client.0.smithi045.stdout:killing divergent 1 2020-03-11T22:44:57.918 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:57.919 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:44:45.807489+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:57.919 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:57.919 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:57.920 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:57.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:44:57.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:57.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:57.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:57.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:57.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:58.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:58.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:44:58.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:58.034 INFO:tasks.workunit.client.0.smithi045.stdout:reviving divergent 1 2020-03-11T22:44:58.377 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:58.378 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:44:45.807489+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:44:58.378 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:44:58.378 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:58.379 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:44:58.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:44:58.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:58.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:58.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:58.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:58.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:58.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:58.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:58.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:58.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:58.396 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:58.397 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:58.397 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:58.397 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:58.397 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:58.398 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:44:58.398 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:44:58.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' 2020-03-11T22:44:58.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:58.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:58.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:58.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:58.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:58.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e7ca7a1c-b3b1-4cc5-a901-65834a3a80e3 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17438/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:58.400 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:58.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:58.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:58.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:58.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:58.403 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:44:58.418 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:58.415+0000 7fec7e30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:58.419 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:58.417+0000 7fec7e30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:58.421 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:58.418+0000 7fec7e30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:58.756 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:58.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:58.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:58.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:58.758 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:44:58.939 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:58.937+0000 7fec7e30cec0 -1 Falling back to public interface 2020-03-11T22:44:59.156 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/51973,v1:127.0.0.1:6819/51973] [v2:127.0.0.1:6820/51973,v1:127.0.0.1:6821/51973] exists,up 5b96c0dd-0926-4a5f-acff-35ff3d1763e2 2020-03-11T22:44:59.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:59.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:59.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:59.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:44:59.787 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:44:59.784+0000 7fec7e30cec0 -1 osd.1 52 log_to_monitors {default=true} 2020-03-11T22:45:19.157 INFO:tasks.workunit.client.0.smithi045.stdout:allowing recovery 2020-03-11T22:45:19.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:45:19.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:19.510 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:19.510 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:01.191051+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:45:19.511 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:45:19.511 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:19.512 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:45:19.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:19.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:45:19.648 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:19.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:19.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:45:19.783 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:19.791 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:19.791 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:45:19.918 INFO:tasks.workunit.client.0.smithi045.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:19.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:45:19.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:19.942 INFO:tasks.workunit.client.0.smithi045.stdout:reading divergent objects 2020-03-11T22:45:20.265 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:20.280 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:01.191051+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:27.061355+0000 0'0 2020-03-11T22:42:27.061355+0000 0 2020-03-11T22:45:20.280 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:45:20.280 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:20.281 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:45:20.281 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:45:20.281 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:45:20.281 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:20.282 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:45:20.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:20.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:45:20.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:20.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:45:20.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:20.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:45:20.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:45:20.420 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:20.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:20.424 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:48.892080+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.424 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:48.892080+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.424 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4559.0:1 2020-03-11T22:42:55.826566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4559.0:1 2020-03-11T22:42:55.826566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4559.0:1 2020-03-11T22:42:55.826566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:45:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4550.0:1 2020-03-11T22:42:54.665027+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4556.0:1 2020-03-11T22:42:55.671238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4550.0:1 2020-03-11T22:42:54.665027+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4556.0:1 2020-03-11T22:42:55.671238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2020-03-11T22:45:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4556.0:1 2020-03-11T22:42:55.671238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:45.774+0000 7fb8c2fc8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:20.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:20.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:45:20.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:45:20.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:45:20.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:45:20.432 INFO:tasks.workunit.client.0.smithi045.stdout:success 2020-03-11T22:45:20.927 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' does not exist 2020-03-11T22:45:20.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:21.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:21.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:21.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:21.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:21.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:21.050 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:21.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:21.051 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:21.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:21.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:21.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.054 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:21.054 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:21.054 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:21.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:21.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:21.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:21.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:45:21.069 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:45:21.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:21.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:21.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:45:21.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:45:21.071 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:21.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:21.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:21.073 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:21.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:21.074 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:21.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:21.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:21.075 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:21.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:21.077 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:21.077 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:21.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:45:21.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:21.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:21.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17438 2020-03-11T22:45:21.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17438 2020-03-11T22:45:21.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:21.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:21.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:45:21.082 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:45:21.082 INFO:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:45:21.189 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:45:21.189 INFO:teuthology.orchestra.run.smithi045:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:45:21.237 INFO:tasks.workunit.client.0.smithi045.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:45:21.245 INFO:tasks.workunit.client.0.smithi045.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:45:21.245 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:45:21.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:45:21.247 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none ' 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:45:21.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:45:21.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:45:21.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:45:21.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:45:21.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:45:21.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:21.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:21.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:21.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:21.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:21.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:21.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:21.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:21.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:21.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52903 2020-03-11T22:45:21.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:21.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:21.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:45:21.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:45:21.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:21.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52903 2020-03-11T22:45:21.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:45:21.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:45:21.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:45:21.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:45:21.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:45:21.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:45:21.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:45:21.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:45:21.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:45:21.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:45:21.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:45:21.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:45:21.602 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:45:21.602 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:21.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:45:21.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:45:21.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:45:21.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:21.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:21.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.765 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:21.766 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52903/ceph-mon.a.asok 2020-03-11T22:45:21.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:21.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52903/ceph-mon.a.asok config get fsid 2020-03-11T22:45:21.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:45:21.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:21.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:21.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:21.851 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:21.852 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52903/ceph-mon.a.asok 2020-03-11T22:45:21.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:21.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52903/ceph-mon.a.asok config get mon_host 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:45:21.942 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:45:21.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:45:22.842 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:45:22.842 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:22.842 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:22.843 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:22.843 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:22.843 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:22.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:22.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:22.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:22.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:45:22.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:22.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:22.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:45:22.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:45:22.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:22.867 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:22.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:22.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:22.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:45:22.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:22.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c29853e2-95ab-4991-9512-c8516e7340c9 2020-03-11T22:45:22.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 c29853e2-95ab-4991-9512-c8516e7340c9' 2020-03-11T22:45:22.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:22.871 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 c29853e2-95ab-4991-9512-c8516e7340c9 2020-03-11T22:45:22.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQACamleH1azNBAALEqLq3d5/crvTVLp6oS8HQ== 2020-03-11T22:45:22.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQACamleH1azNBAALEqLq3d5/crvTVLp6oS8HQ=="}' 2020-03-11T22:45:22.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c29853e2-95ab-4991-9512-c8516e7340c9 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:45:23.253 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:23.262 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:45:23.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACamleH1azNBAALEqLq3d5/crvTVLp6oS8HQ== --osd-uuid c29853e2-95ab-4991-9512-c8516e7340c9 2020-03-11T22:45:23.284 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.280+0000 7f59d5b93ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:23.285 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.282+0000 7f59d5b93ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:23.287 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.283+0000 7f59d5b93ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:23.287 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.284+0000 7f59d5b93ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:23.287 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.284+0000 7f59d5b93ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:23.288 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.284+0000 7f59d5b93ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:23.393 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:23.390+0000 7f59d5b93ec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:45:26.570 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:45:26.570 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:26.571 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:45:26.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:45:26.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:27.050 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:45:27.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:45:27.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:27.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:27.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:27.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:27.069 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:27.066+0000 7fe94f12cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:27.071 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:27.068+0000 7fe94f12cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:27.073 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:27.069+0000 7fe94f12cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:27.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:45:27.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:27.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:27.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:27.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:27.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:27.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:27.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:27.421 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:27.421 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:27.590 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:27.586+0000 7fe94f12cec0 -1 Falling back to public interface 2020-03-11T22:45:27.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:28.457 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:28.453+0000 7fe94f12cec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:45:28.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:28.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:28.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:28.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:28.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:28.787 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:45:29.160 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53332,v1:127.0.0.1:6803/53332] [v2:127.0.0.1:6804/53332,v1:127.0.0.1:6805/53332] exists,up c29853e2-95ab-4991-9512-c8516e7340c9 2020-03-11T22:45:29.160 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 a848afea-0e8d-4c8f-af40-5a85ed154e4b 2020-03-11T22:45:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:45:29.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:45:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:45:29.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:29.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:29.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:29.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:29.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:29.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:29.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a848afea-0e8d-4c8f-af40-5a85ed154e4b 2020-03-11T22:45:29.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 a848afea-0e8d-4c8f-af40-5a85ed154e4b' 2020-03-11T22:45:29.168 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:29.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAJamleYYUVChAAUzrFb9fn77K22AI8nMGsHA== 2020-03-11T22:45:29.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAJamleYYUVChAAUzrFb9fn77K22AI8nMGsHA=="}' 2020-03-11T22:45:29.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a848afea-0e8d-4c8f-af40-5a85ed154e4b -i td/ec-error-rollforward/1/new.json 2020-03-11T22:45:29.698 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:45:29.708 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:45:29.709 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJamleYYUVChAAUzrFb9fn77K22AI8nMGsHA== --osd-uuid a848afea-0e8d-4c8f-af40-5a85ed154e4b 2020-03-11T22:45:29.730 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.726+0000 7f95b86f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:29.731 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.728+0000 7f95b86f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.729+0000 7f95b86f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:29.734 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.730+0000 7f95b86f4ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:29.734 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.730+0000 7f95b86f4ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:29.734 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.730+0000 7f95b86f4ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:29.781 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:29.778+0000 7f95b86f4ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:45:33.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:45:33.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:33.150 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:45:33.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:45:33.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:33.631 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:45:33.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:33.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:33.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:33.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:33.632 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:45:33.650 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:33.647+0000 7f4c12596ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:33.652 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:33.648+0000 7f4c12596ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:33.653 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:33.649+0000 7f4c12596ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:33.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:34.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:34.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:34.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:34.000 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:34.169 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:34.165+0000 7f4c12596ec0 -1 Falling back to public interface 2020-03-11T22:45:34.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:34.978 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:34.974+0000 7f4c12596ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:45:35.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:35.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:35.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:35.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:35.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:35.361 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:45:35.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:36.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:36.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:36.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:45:36.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:36.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:36.742 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:45:37.120 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53688,v1:127.0.0.1:6811/53688] [v2:127.0.0.1:6812/53688,v1:127.0.0.1:6813/53688] exists,up a848afea-0e8d-4c8f-af40-5a85ed154e4b 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:37.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:37.123 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:37.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:37.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:37.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:37.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:45:37.126 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:37.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 2020-03-11T22:45:37.127 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51' 2020-03-11T22:45:37.127 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:37.127 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 2020-03-11T22:45:37.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQARamle6TzqBxAANxPExWW+mwnpzo30UEvr+g== 2020-03-11T22:45:37.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQARamle6TzqBxAANxPExWW+mwnpzo30UEvr+g=="}' 2020-03-11T22:45:37.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:45:37.643 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:45:37.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:45:37.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARamle6TzqBxAANxPExWW+mwnpzo30UEvr+g== --osd-uuid 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 2020-03-11T22:45:37.674 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.671+0000 7f6525943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.676 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.672+0000 7f6525943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.677 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.673+0000 7f6525943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.677 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.674+0000 7f6525943ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:37.677 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.674+0000 7f6525943ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:37.677 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.674+0000 7f6525943ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:37.763 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:37.759+0000 7f6525943ec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:45:41.182 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:45:41.182 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:41.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:45:41.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:41.184 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:45:41.667 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:45:41.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:45:41.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:41.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:41.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:41.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:41.686 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:41.682+0000 7f0f6f0baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:41.687 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:41.684+0000 7f0f6f0baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:41.689 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:41.686+0000 7f0f6f0baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:42.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:45:42.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:42.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:42.205 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:42.201+0000 7f0f6f0baec0 -1 Falling back to public interface 2020-03-11T22:45:42.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:43.032 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:43.028+0000 7f0f6f0baec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:45:43.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:43.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:43.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:43.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:43.409 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:43.409 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:45:43.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:44.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:44.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:44.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:45:44.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:44.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:44.784 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:45:45.179 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54071,v1:127.0.0.1:6819/54071] [v2:127.0.0.1:6820/54071,v1:127.0.0.1:6821/54071] exists,up 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 2020-03-11T22:45:45.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:45.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:45.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:45.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:45.182 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:45.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:45.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:45:45.185 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:45.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=69827025-c864-4713-91fc-3807c1db1cf4 2020-03-11T22:45:45.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 69827025-c864-4713-91fc-3807c1db1cf4' 2020-03-11T22:45:45.186 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:45.186 INFO:tasks.workunit.client.0.smithi045.stdout:add osd3 69827025-c864-4713-91fc-3807c1db1cf4 2020-03-11T22:45:45.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAZamlebJdkCxAAwbLu8xymLd+olZD151MnPw== 2020-03-11T22:45:45.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAZamlebJdkCxAAwbLu8xymLd+olZD151MnPw=="}' 2020-03-11T22:45:45.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 69827025-c864-4713-91fc-3807c1db1cf4 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:45:45.656 INFO:tasks.workunit.client.0.smithi045.stdout:3 2020-03-11T22:45:45.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:45:45.667 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZamlebJdkCxAAwbLu8xymLd+olZD151MnPw== --osd-uuid 69827025-c864-4713-91fc-3807c1db1cf4 2020-03-11T22:45:45.688 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.684+0000 7fb77bebfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.689 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.686+0000 7fb77bebfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.690 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.687+0000 7fb77bebfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.691 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.687+0000 7fb77bebfec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:45.691 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.687+0000 7fb77bebfec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:45.691 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.687+0000 7fb77bebfec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:45.762 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:45.759+0000 7fb77bebfec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:45:49.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:45:49.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:49.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:45:49.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:49.151 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd3 key to auth repository 2020-03-11T22:45:49.627 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.3 2020-03-11T22:45:49.627 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:45:49.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:49.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:49.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:49.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:49.646 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:49.642+0000 7f1e75d5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:49.647 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:49.644+0000 7f1e75d5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:49.649 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:49.645+0000 7f1e75d5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:45:50.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:50.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:45:50.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:50.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:50.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:50.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:50.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:50.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:50.011 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:50.170 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:50.166+0000 7f1e75d5aec0 -1 Falling back to public interface 2020-03-11T22:45:50.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:50.943 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:50.941+0000 7f1e75d5aec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:51.399 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:45:51.794 INFO:tasks.workunit.client.0.smithi045.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54462,v1:127.0.0.1:6827/54462] [v2:127.0.0.1:6828/54462,v1:127.0.0.1:6829/54462] exists,up 69827025-c864-4713-91fc-3807c1db1cf4 2020-03-11T22:45:51.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:51.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:51.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:51.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:45:52.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:45:53.172 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'ec' already exists 2020-03-11T22:45:53.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:45:54.350 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:45:54.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54071 2020-03-11T22:45:54.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54718 2020-03-11T22:45:54.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:45:54.351 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:45:55.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54741' 2020-03-11T22:45:55.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:45:55.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54742' 2020-03-11T22:45:55.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:45:55.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54743' 2020-03-11T22:45:55.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:45:55.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:45:56.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:45:56.356 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:56.356 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:56.356 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:56.356 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:56.356 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:56.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:56.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54718 54741 54742 54743 2020-03-11T22:45:56.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:45:56.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:45:56.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:56.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:56.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:56.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:56.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:56.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:56.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:45:56.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:45:56.474 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:56.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:56.490 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:56.488+0000 7f214bb08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:56.492 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:56.490+0000 7f214bb08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:56.494 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:56.492+0000 7f214bb08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:56.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:56.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:56.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:56.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:56.843 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:57.010 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:57.007+0000 7f214bb08ec0 -1 Falling back to public interface 2020-03-11T22:45:57.220 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53332,v1:127.0.0.1:6803/53332] [v2:127.0.0.1:6804/53332,v1:127.0.0.1:6805/53332] exists,up c29853e2-95ab-4991-9512-c8516e7340c9 2020-03-11T22:45:57.221 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:45:57.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:57.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:57.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:57.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:45:57.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:57.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:57.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:57.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:57.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:57.225 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:57.225 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:57.225 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:57.225 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:57.225 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:57.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:57.227 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:45:57.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:57.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:57.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:57.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:57.237 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:57.235+0000 7fc6f12bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.239 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:57.237+0000 7fc6f12bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.241 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:57.238+0000 7fc6f12bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:57.760 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:57.757+0000 7fc6f12bdec0 -1 Falling back to public interface 2020-03-11T22:45:58.009 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.006+0000 7f214bb08ec0 -1 osd.0 26 log_to_monitors {default=true} 2020-03-11T22:45:58.039 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53688,v1:127.0.0.1:6811/53688] [v2:127.0.0.1:6812/53688,v1:127.0.0.1:6813/53688] exists,up a848afea-0e8d-4c8f-af40-5a85ed154e4b 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:58.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:58.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:58.042 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:45:58.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:58.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:58.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:58.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:58.047 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:45:58.059 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.057+0000 7f86f2d8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:58.061 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.059+0000 7f86f2d8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:58.062 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.059+0000 7f86f2d8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:58.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:45:58.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:58.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:58.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:58.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:58.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:58.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:58.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:58.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:58.473 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:45:58.580 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.578+0000 7f86f2d8fec0 -1 Falling back to public interface 2020-03-11T22:45:58.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:58.895 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:58.893+0000 7fc6f12bdec0 -1 osd.1 26 log_to_monitors {default=true} 2020-03-11T22:45:59.832 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:45:59.830+0000 7f86f2d8fec0 -1 osd.2 25 log_to_monitors {default=true} 2020-03-11T22:45:59.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:59.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:59.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:59.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:59.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:59.876 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:46:00.379 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,25) [v2:127.0.0.1:6818/55108,v1:127.0.0.1:6819/55108] [v2:127.0.0.1:6820/55108,v1:127.0.0.1:6821/55108] exists,up 81ddea5a-f7cb-4244-8ea9-7bdf9a68dc51 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:00.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:00.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:00.382 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' 2020-03-11T22:46:00.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:00.384 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:00.385 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:00.385 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:46:00.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:46:00.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=58c39f22-9537-4751-b316-0c3103841702 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52903/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:00.386 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:46:00.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:46:00.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:00.387 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:00.387 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:00.387 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.3 2020-03-11T22:46:00.399 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:00.396+0000 7f7661235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:00.401 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:00.398+0000 7f7661235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:00.402 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:00.400+0000 7f7661235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:00.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:46:00.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:00.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:00.745 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:46:00.922 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:00.920+0000 7f7661235ec0 -1 Falling back to public interface 2020-03-11T22:46:01.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:01.812 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:01.809+0000 7f7661235ec0 -1 osd.3 26 log_to_monitors {default=true} 2020-03-11T22:46:02.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:02.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:02.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:02.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:02.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:02.118 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi045.stdout:osd.3 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/55374,v1:127.0.0.1:6827/55374] [v2:127.0.0.1:6828/55374,v1:127.0.0.1:6829/55374] exists,up 69827025-c864-4713-91fc-3807c1db1cf4 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:02.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:02.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:02.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:02.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:02.676 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:03.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:03.037 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:46:03.037 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:46:03.037 INFO:tasks.workunit.client.0.smithi045.stderr:3' 2020-03-11T22:46:03.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:03.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.038 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:03.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084292 2020-03-11T22:46:03.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292' 2020-03-11T22:46:03.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.165 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:03.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:46:03.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179' 2020-03-11T22:46:03.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.292 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:03.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:46:03.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-141733920771' 2020-03-11T22:46:03.419 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:46:03.544 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:46:03.545 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-141733920771 3-150323855363' 2020-03-11T22:46:03.545 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:03.545 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-120259084292 2020-03-11T22:46:03.545 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:03.546 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:03.549 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-120259084292 2020-03-11T22:46:03.550 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:03.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084292 2020-03-11T22:46:03.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2020-03-11T22:46:03.550 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:03.550 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 120259084292 2020-03-11T22:46:03.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 120259084292 2020-03-11T22:46:03.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:04.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:04.910 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:05.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084292 -lt 120259084292 2020-03-11T22:46:05.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:05.286 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:05.286 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:05.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:05.288 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:05.288 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:05.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:46:05.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2020-03-11T22:46:05.289 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 133143986179 2020-03-11T22:46:05.290 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:05.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986180 -lt 133143986179 2020-03-11T22:46:05.654 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:05.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-141733920771 2020-03-11T22:46:05.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:05.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:05.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-141733920771 2020-03-11T22:46:05.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:05.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:46:05.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-03-11T22:46:05.658 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 141733920771 2020-03-11T22:46:05.659 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:06.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:46:06.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:06.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-150323855363 2020-03-11T22:46:06.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:06.058 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:46:06.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-150323855363 2020-03-11T22:46:06.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:06.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:46:06.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2020-03-11T22:46:06.060 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 150323855363 2020-03-11T22:46:06.061 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:46:06.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:46:06.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:06.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:06.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:06.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:06.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:06.911 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:07.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:07.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:07.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:07.262 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:07.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:07.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:07.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:07.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:46:07.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:07.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:07.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:07.732 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:07.732 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:07.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:07.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:07.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:07.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:07.840 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:07.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:07.840 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:07.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:07.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:07.842 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:07.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:07.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:07.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:07.844 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:07.844 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:07.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:07.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:07.851 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:07.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:07.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:46:07.852 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52903 2020-03-11T22:46:07.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:07.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:07.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:46:07.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:46:07.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:07.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:07.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:07.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:07.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:07.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:07.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:07.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:07.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:07.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:07.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:07.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:07.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:07.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:07.860 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:46:07.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:07.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:07.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:07.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52903 2020-03-11T22:46:07.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52903 2020-03-11T22:46:07.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:07.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:07.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:46:07.864 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:46:07.864 INFO:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:46:07.939 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:46:07.939 INFO:teuthology.orchestra.run.smithi045:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:46:07.995 INFO:tasks.workunit.client.0.smithi045.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:46:08.004 INFO:tasks.workunit.client.0.smithi045.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:46:08.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:08.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none ' 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:46:08.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:46:08.008 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:46:08.009 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_backfill_priority' 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:46:08.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:08.012 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:08.013 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:08.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:08.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:08.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:08.015 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:08.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56115 2020-03-11T22:46:08.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:08.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:08.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:46:08.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:08.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:08.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56115 2020-03-11T22:46:08.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:08.021 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:46:08.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.340 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:08.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:08.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:08.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:08.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:08.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:08.501 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:08.502 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:08.503 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.504 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:08.504 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-mon.a.asok 2020-03-11T22:46:08.504 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:08.504 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56115/ceph-mon.a.asok config get fsid 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:08.596 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-mon.a.asok 2020-03-11T22:46:08.597 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:08.598 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56115/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:08.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:46:08.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:46:08.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:08.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:08.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:08.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:46:08.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:09.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:09.602 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:09.602 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:09.602 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:09.602 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:09.602 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:09.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:09.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:09.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:09.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:46:09.626 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:46:09.628 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:46:09.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:09.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:46:09.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:09.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:09.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:09.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:09.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:46:09.635 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:09.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:09.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:09.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:09.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:46:09.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:46:09.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:09.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:09.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:09.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:09.639 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:09.639 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:09.639 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:09.639 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:09.640 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:09.640 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:09.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:09.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:09.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:09.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:09.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:09.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:09.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:09.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:09.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:09.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:09.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:09.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:46:09.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:09.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=328088fb-5a31-42b8-a2bd-c238b2c18da7 2020-03-11T22:46:09.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 328088fb-5a31-42b8-a2bd-c238b2c18da7' 2020-03-11T22:46:09.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:09.644 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 328088fb-5a31-42b8-a2bd-c238b2c18da7 2020-03-11T22:46:09.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAxamleLp97JhAAEDwPSUbJCUUCrVIQViWvQg== 2020-03-11T22:46:09.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAxamleLp97JhAAEDwPSUbJCUUCrVIQViWvQg=="}' 2020-03-11T22:46:09.649 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 328088fb-5a31-42b8-a2bd-c238b2c18da7 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:46:09.959 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:46:09.969 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:46:09.970 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAxamleLp97JhAAEDwPSUbJCUUCrVIQViWvQg== --osd-uuid 328088fb-5a31-42b8-a2bd-c238b2c18da7 2020-03-11T22:46:09.996 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.993+0000 7fb7e97cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:09.998 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.995+0000 7fb7e97cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:10.000 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.997+0000 7fb7e97cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:10.001 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.998+0000 7fb7e97cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:10.001 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.998+0000 7fb7e97cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:10.001 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:09.998+0000 7fb7e97cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:10.100 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:10.097+0000 7fb7e97cbec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:46:13.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:46:13.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:13.167 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:46:13.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:13.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:13.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:13.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:13.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:13.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:13.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:13.703 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:46:13.726 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:13.723+0000 7f3f5e18dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:13.759 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:13.757+0000 7f3f5e18dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:13.760 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:13.758+0000 7f3f5e18dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:14.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:14.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:14.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:14.072 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:14.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:14.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:14.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:14.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:14.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:14.074 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:46:14.276 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:14.273+0000 7f3f5e18dec0 -1 Falling back to public interface 2020-03-11T22:46:14.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:15.058 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:15.055+0000 7f3f5e18dec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:15.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:15.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:15.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:15.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:15.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:15.491 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:46:15.881 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56547,v1:127.0.0.1:6803/56547] [v2:127.0.0.1:6804/56547,v1:127.0.0.1:6805/56547] exists,up 328088fb-5a31-42b8-a2bd-c238b2c18da7 2020-03-11T22:46:15.882 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 1368e2fe-f65f-4fec-b21a-4a7ef25c5879 2020-03-11T22:46:15.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:15.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:15.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:15.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:46:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:46:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:46:15.888 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1368e2fe-f65f-4fec-b21a-4a7ef25c5879 2020-03-11T22:46:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 1368e2fe-f65f-4fec-b21a-4a7ef25c5879' 2020-03-11T22:46:15.888 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:15.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA3amleer1kNRAABFQkMMetpLEQ8wir/xEC0A== 2020-03-11T22:46:15.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA3amleer1kNRAABFQkMMetpLEQ8wir/xEC0A=="}' 2020-03-11T22:46:15.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1368e2fe-f65f-4fec-b21a-4a7ef25c5879 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:46:16.429 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:46:16.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:46:16.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3amleer1kNRAABFQkMMetpLEQ8wir/xEC0A== --osd-uuid 1368e2fe-f65f-4fec-b21a-4a7ef25c5879 2020-03-11T22:46:16.466 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.463+0000 7fdfeb03dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:16.497 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.495+0000 7fdfeb03dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:16.499 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.497+0000 7fdfeb03dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:16.500 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.497+0000 7fdfeb03dec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:16.500 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.497+0000 7fdfeb03dec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:16.500 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.497+0000 7fdfeb03dec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:16.597 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:16.595+0000 7fdfeb03dec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:46:19.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:46:19.708 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:19.709 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:19.709 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:19.709 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:46:20.203 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:46:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:20.227 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:20.224+0000 7f12a8d26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:20.229 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:20.226+0000 7f12a8d26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:20.231 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:20.227+0000 7f12a8d26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:20.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:20.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:20.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:20.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:20.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:20.579 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:20.579 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:20.579 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:20.579 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:20.579 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:46:20.746 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:20.742+0000 7f12a8d26ec0 -1 Falling back to public interface 2020-03-11T22:46:20.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:21.529 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:46:21.525+0000 7f12a8d26ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:46:21.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:21.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:21.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:21.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:21.948 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:21.948 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:46:22.329 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56902,v1:127.0.0.1:6811/56902] [v2:127.0.0.1:6812/56902,v1:127.0.0.1:6813/56902] exists,up 1368e2fe-f65f-4fec-b21a-4a7ef25c5879 2020-03-11T22:46:22.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:22.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:22.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:22.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:22.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:22.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:46:22.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:46:22.952 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test1' already exists 2020-03-11T22:46:22.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:23.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:46:24.774 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 2 2020-03-11T22:46:24.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:24.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:46:24.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:46:25.417 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test2' already exists 2020-03-11T22:46:25.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:26.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:46:27.244 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 2 size to 2 2020-03-11T22:46:27.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:27.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:46:27.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:46:27.764 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test3' already exists 2020-03-11T22:46:27.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:28.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:46:29.581 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 3 size to 2 2020-03-11T22:46:29.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:46:34.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:34.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:34.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:34.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:34.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:34.593 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:34.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:34.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:34.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:34.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:34.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:35.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:35.065 INFO:tasks.workunit.client.0.smithi045.stderr:1' 2020-03-11T22:46:35.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:35.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:35.066 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:35.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:46:35.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:46:35.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:35.200 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:35.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:46:35.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:46:35.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:35.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:35.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:35.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:35.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:35.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:35.337 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:46:35.337 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:46:35.337 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:46:35.338 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:35.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:46:35.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:36.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:36.706 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:37.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:46:37.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:38.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:46:38.086 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:38.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836486 2020-03-11T22:46:38.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:38.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:46:38.480 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:38.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:38.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:46:38.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:38.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:46:38.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:46:38.483 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:46:38.483 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:38.852 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:46:38.852 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:38.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:38.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:39.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:46:39.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:39.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:39.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:46:39.683 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:39.683 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:39.683 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:40.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:46:40.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:40.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:40.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:46:40.498 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:40.499 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:30.296826+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:27.576534+0000 0'0 2020-03-11T22:46:27.576534+0000 0 2020-03-11T22:46:40.499 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:24.594095+0000 0'0 27:28 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:22.740736+0000 0'0 2020-03-11T22:46:22.740736+0000 0 2020-03-11T22:46:40.499 INFO:tasks.workunit.client.0.smithi045.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:27.419216+0000 0'0 26:23 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:46:25.152988+0000 0'0 2020-03-11T22:46:25.152988+0000 0 2020-03-11T22:46:40.499 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:46:40.500 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:40.500 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:46:40.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:46:40.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:46:40.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:46:40.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:46:40.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:46:40.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:46:40.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:46:40.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:46:40.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:46:40.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:46:40.511 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:40.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:40.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:46:40.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:40.891 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:40.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:46:40.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:40.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:46:40.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:46:40.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:46:40.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:46:40.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:41.253 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:41.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:46:41.254 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:46:41.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:46:41.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:46:41.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:46:41.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:46:41.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:46:41.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:46:41.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:41.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:46:41.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:46:41.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:46:41.261 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:46:41.786 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test3' does not exist 2020-03-11T22:46:41.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:46:41.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:46:41.795 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:46:41.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:46:41.797 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:46:41.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:46:41.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:46:42.478 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 1 2020-03-11T22:46:42.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:46:43.288 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:46:43.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:46:43.998 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 2 size to 1 2020-03-11T22:46:44.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:46:44.632 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:46:44.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:44.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:44.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:44.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:44.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:44.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:44.642 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:44.643 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:44.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:44.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:44.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:44.734 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:44.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:44.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:44.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:44.735 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:45.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:45.119 INFO:tasks.workunit.client.0.smithi045.stderr:1' 2020-03-11T22:46:45.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:45.120 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.120 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:45.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:46:45.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:46:45.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:45.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:46:45.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-03-11T22:46:45.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:45.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:45.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:45.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:45.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:45.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:45.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:46:45.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:46:45.394 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:46:45.394 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:45.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:46:45.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:46.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:46.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:47.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:46:47.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:48.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:46:48.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:48.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836490 2020-03-11T22:46:48.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:48.526 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:48.526 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:48.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:48.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:48.528 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:48.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:46:48.529 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:46:48.530 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:46:48.530 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:48.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:46:48.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:48.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:48.936 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:49.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:46:49.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:49.416 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:49.417 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:49.417 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:49.417 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:49.417 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:49.418 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:49.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:46:49.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:49.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:49.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:50.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:46:50.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:50.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:50.250 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:46:50.296 INFO:tasks.workunit.client.0.smithi045.stderr:10+0 records in 2020-03-11T22:46:50.296 INFO:tasks.workunit.client.0.smithi045.stderr:10+0 records out 2020-03-11T22:46:50.296 INFO:tasks.workunit.client.0.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0456574 s, 230 MB/s 2020-03-11T22:46:50.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:46:50.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:46:50.297 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:46:50.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:50.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:46:50.546 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:50.546 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:46:50.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:50.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:46:51.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:51.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:46:51.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:51.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:46:51.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:51.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:46:51.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:51.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:46:52.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:52.247 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:46:52.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:52.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:46:52.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:52.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.281 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.530 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.530 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.823 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.823 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.548 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.549 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.615 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.621 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.066 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.177 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:47:04.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:47:04.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:47:04.468 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:47:04.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:47:04.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:47:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.013 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:47:05.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:47:05.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:47:05.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.847 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:47:06.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:47:06.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:47:06.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:47:06.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:47:07.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:47:07.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:47:07.922 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.922 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.748 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.808 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.092 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.092 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.652 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:47:18.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:47:18.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:47:18.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:47:18.656 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:47:18.656 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:47:18.656 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:47:18.656 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:47:18.657 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:18.657 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:47:19.045 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:47:19.046 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:19.046 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:47:19.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:47:19.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:46:42.521303+0000 37'150 37:193 [1] 1 [1] 1 0'0 2020-03-11T22:46:22.740736+0000 0'0 2020-03-11T22:46:22.740736+0000 0 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi045.stdout:2.0 42 0 0 0 0 440401920 0 0 26 26 active+clean 2020-03-11T22:46:44.019702+0000 37'126 37:164 [0] 0 [0] 0 0'0 2020-03-11T22:46:25.152988+0000 0'0 2020-03-11T22:46:25.152988+0000 0 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:20.471 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 2 2020-03-11T22:47:20.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:21.482 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 2 size to 2 2020-03-11T22:47:21.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:47:26.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:47:26.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:47:26.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:47:26.496 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:26.496 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:26.497 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:47:26.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:47:26.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:26.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:47:26.607 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 2020-03-11T22:47:26.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:47:26.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:26.607 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:47:26.608 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 152, 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:47:26.609 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "2.0", 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 153, 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:47:26.610 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.611 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:47:26.611 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.611 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:47:26.612 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:47:26.612 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:47:26.612 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.0.asok 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:26.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:47:26.713 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 2020-03-11T22:47:26.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:47:26.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:47:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "2.0", 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 153, 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:47:26.715 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:47:26.716 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 152, 2020-03-11T22:47:26.717 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:47:26.717 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.717 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:47:26.717 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:47:26.717 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:47:26.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:26.719 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:26.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:26.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:26.721 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:26.721 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:26.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:26.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:26.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:26.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:26.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:26.727 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:26.727 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:26.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:26.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:26.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:26.730 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:26.731 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:26.731 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:26.731 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:26.734 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:26.734 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:26.735 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:26.735 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:26.737 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:26.737 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:26.737 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:26.738 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:26.738 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:26.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:26.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:26.742 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:26.743 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:26.743 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:26.743 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:26.743 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:26.743 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:26.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:26.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:26.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:26.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:26.836 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:27.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:27.201 INFO:tasks.workunit.client.0.smithi045.stderr:1' 2020-03-11T22:47:27.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:27.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:27.201 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:27.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836501 2020-03-11T22:47:27.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836501' 2020-03-11T22:47:27.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:27.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:27.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672979 2020-03-11T22:47:27.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2020-03-11T22:47:27.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:27.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836501 2020-03-11T22:47:27.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:27.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:27.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836501 2020-03-11T22:47:27.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:27.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836501 2020-03-11T22:47:27.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2020-03-11T22:47:27.565 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836501 2020-03-11T22:47:27.566 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:28.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836501 2020-03-11T22:47:28.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:29.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:29.065 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:29.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836501 -lt 21474836501 2020-03-11T22:47:29.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:29.669 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672979 2020-03-11T22:47:29.669 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:29.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:29.671 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672979 2020-03-11T22:47:29.671 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:29.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672979 2020-03-11T22:47:29.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2020-03-11T22:47:29.672 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672979 2020-03-11T22:47:29.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:30.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672980 -lt 42949672979 2020-03-11T22:47:30.077 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:30.077 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:30.077 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:30.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:47:30.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:30.638 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:30.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:31.000 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:31.000 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:31.001 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:31.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:31.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:31.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:31.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:31.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:31.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:31.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:31.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:31.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:32.128 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:32.128 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:32.129 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:32.129 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:32.624 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:32.624 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:32.624 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:32.624 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:32.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:32.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:32.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:32.625 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:32.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:33.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20782034 2020-03-11T22:47:33.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20782034 '!=' null 2020-03-11T22:47:33.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:33.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:33.119 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:33.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:33.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:33.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:33.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:33.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:33.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:33.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:33.221 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:33.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:33.584 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:33.584 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:33.584 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:34.063 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:34.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:34.064 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:34.064 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:34.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31059938 2020-03-11T22:47:34.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31059938 '!=' null 2020-03-11T22:47:34.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:34.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:34.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:34.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:34.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:34.688 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:35.047 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:35.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:35.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:35.048 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:35.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:35.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:35.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:35.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:35.748 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:35.748 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:35.748 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:35.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:35.749 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:36.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27082846 2020-03-11T22:47:36.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27082846 '!=' null 2020-03-11T22:47:36.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:36.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:36.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:36.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:36.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:36.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:36.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:36.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:36.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:36.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:36.446 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:36.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:36.829 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:36.830 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:36.830 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:37.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:37.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:37.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:38.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198944 2020-03-11T22:47:38.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198944 '!=' null 2020-03-11T22:47:38.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:38.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:38.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:38.254 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:38.254 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:38.257 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:38.257 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:38.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:38.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:38.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:38.258 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:38.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:38.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:38.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:38.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:39.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:39.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:39.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:39.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:39.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:39.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:39.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:39.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:39.143 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:39.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946292 2020-03-11T22:47:39.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946292 '!=' null 2020-03-11T22:47:39.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:39.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:39.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:39.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:39.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:39.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:40.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:40.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:40.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:40.175 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:40.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:40.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:40.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:40.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:41.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946292 2020-03-11T22:47:41.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946292 '!=' null 2020-03-11T22:47:41.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:41.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:41.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:41.304 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:41.304 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:41.305 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:41.305 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:41.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:41.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:41.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:41.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:41.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:41.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:41.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:41.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:42.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:42.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:42.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:42.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:42.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:42.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:42.146 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:42.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:42.146 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:42.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26209653 2020-03-11T22:47:42.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26209653 '!=' null 2020-03-11T22:47:42.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:42.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:42.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:42.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:42.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:42.985 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:42.985 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:42.985 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:42.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:42.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:42.986 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:43.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:43.346 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:43.346 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:43.347 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:43.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:43.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:43.873 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:43.873 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:44.678 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41062056 2020-03-11T22:47:44.678 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41062056 '!=' null 2020-03-11T22:47:44.678 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:44.678 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:44.678 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:44.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:44.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:44.780 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:44.780 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:44.780 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:44.780 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:44.780 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:44.781 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:45.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:45.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:45.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:45.145 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:45.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:45.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:45.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:45.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:45.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:45.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:45.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:45.774 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:45.775 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:46.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957429 2020-03-11T22:47:46.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957429 '!=' null 2020-03-11T22:47:46.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:46.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:46.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:46.546 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:46.546 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:46.547 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:46.547 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:46.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:46.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:46.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:46.548 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:46.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:46.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:46.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:46.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:47.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:47.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:47.508 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:47.508 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:48.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34073263 2020-03-11T22:47:48.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34073263 '!=' null 2020-03-11T22:47:48.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:48.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:48.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:48.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:48.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:48.177 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:48.177 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:48.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:48.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:48.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:48.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:48.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:48.538 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:48.539 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:48.539 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:49.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:49.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:49.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:49.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:49.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:49.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:49.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:49.133 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:49.133 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:49.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34073138 2020-03-11T22:47:49.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34073138 '!=' null 2020-03-11T22:47:49.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:49.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:49.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:49.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:49.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:49.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:49.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:49.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:49.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:49.868 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:49.869 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:50.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:50.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:50.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:50.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:50.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:50.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:50.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:50.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:50.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:50.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:50.745 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:50.745 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:50.745 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:51.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34073138 2020-03-11T22:47:51.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34073138 '!=' null 2020-03-11T22:47:51.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:51.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:51.277 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:51.377 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:51.377 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:51.378 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:51.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:51.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:51.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:51.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:51.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:51.736 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:51.737 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:51.737 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:51.737 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:52.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:52.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:52.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:52.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:52.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:52.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:52.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:52.239 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:52.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27083377 2020-03-11T22:47:52.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27083377 '!=' null 2020-03-11T22:47:52.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:52.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:52.755 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:52.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:52.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:52.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:52.856 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:52.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:52.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:52.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:52.857 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:53.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:53.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:53.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:53.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:53.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:53.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:53.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:53.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:54.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451623 2020-03-11T22:47:54.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451623 '!=' null 2020-03-11T22:47:54.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:54.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:54.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:54.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:54.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:54.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:54.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:54.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:54.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:54.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:54.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:54.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:54.791 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:54.791 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:54.791 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:55.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:55.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:55.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:55.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:55.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:55.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:55.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:55.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:55.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:55.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41936913 2020-03-11T22:47:55.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41936913 '!=' null 2020-03-11T22:47:55.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:55.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:55.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:55.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:55.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:55.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:55.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:55.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:55.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:55.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:55.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:56.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:56.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:56.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:56.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:56.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:56.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:56.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:56.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:56.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:56.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:56.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:56.691 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:56.691 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:57.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41936913 2020-03-11T22:47:57.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41936913 '!=' null 2020-03-11T22:47:57.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:57.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:57.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:57.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:57.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:57.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:57.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:57.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:57.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:57.308 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:57.308 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:57.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:57.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:57.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:57.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:58.135 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:47:58.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:47:58.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:58.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:47:58.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:58.136 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:58.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:58.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:58.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:58.592 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:58.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:58.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:58.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:59.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:47:59.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:47:59.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:59.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:59.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:59.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:59.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:59.068 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:59.069 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=6989454 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 6989454 '!=' null 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:00.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:48:00.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:00.025 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:00.026 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:00.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:48:00.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:48:00.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:48:00.496 INFO:tasks.workunit.client.0.smithi045.stdout:TEST PASSED 2020-03-11T22:48:01.017 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test1' does not exist 2020-03-11T22:48:01.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:48:01.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:48:01.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:48:01.562 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test2' does not exist 2020-03-11T22:48:01.570 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:48:01.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:01.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:01.572 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:01.572 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:01.572 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:01.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:01.681 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:01.681 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:01.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:01.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:01.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:01.684 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:01.684 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:01.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:01.685 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:01.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:01.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:01.686 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:01.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:01.688 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:01.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:01.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:01.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:01.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:01.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:01.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:01.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:01.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:01.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56115 2020-03-11T22:48:01.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:01.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:01.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:48:01.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:48:01.704 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:01.705 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:01.706 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:01.706 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:01.706 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:01.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:01.707 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:01.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:01.707 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:01.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:01.708 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:01.708 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:01.709 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:01.709 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:01.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:01.710 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:01.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:01.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:01.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:01.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:01.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:01.712 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:01.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56115 2020-03-11T22:48:01.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:01.713 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:01.714 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:48:01.714 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:48:01.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:01.715 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:01.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56115 2020-03-11T22:48:01.716 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:48:01.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:48:01.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:48:01.719 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:48:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:48:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:48:01.720 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:48:02.036 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:48:02.036 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:02.036 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:02.036 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:02.036 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:02.037 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:02.037 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:02.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:48:02.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:48:02.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:48:02.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:02.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:02.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:48:02.213 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:02.215 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-mon.a.asok 2020-03-11T22:48:02.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:02.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56115/ceph-mon.a.asok config get fsid 2020-03-11T22:48:02.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:48:02.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:02.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:02.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:48:02.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:02.308 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-mon.a.asok 2020-03-11T22:48:02.309 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:02.309 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56115/ceph-mon.a.asok config get mon_host 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:48:02.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:48:02.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:02.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:02.928 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:02.929 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:02.929 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:02.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:02.930 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:02.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:48:02.955 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:48:02.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:48:02.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:02.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:48:02.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:02.960 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:02.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:02.962 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:02.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:02.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:02.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:48:02.965 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:02.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=43088468-da02-422f-86f4-3b7c265e3bc4 2020-03-11T22:48:02.966 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 43088468-da02-422f-86f4-3b7c265e3bc4' 2020-03-11T22:48:02.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:02.966 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 43088468-da02-422f-86f4-3b7c265e3bc4 2020-03-11T22:48:02.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCiamleKAoUOhAA8sI7aJmEjuHsQ9Q9pmdbBQ== 2020-03-11T22:48:02.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCiamleKAoUOhAA8sI7aJmEjuHsQ9Q9pmdbBQ=="}' 2020-03-11T22:48:02.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 43088468-da02-422f-86f4-3b7c265e3bc4 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:48:03.371 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:03.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:48:03.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCiamleKAoUOhAA8sI7aJmEjuHsQ9Q9pmdbBQ== --osd-uuid 43088468-da02-422f-86f4-3b7c265e3bc4 2020-03-11T22:48:03.408 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.405+0000 7fda97dbbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.409 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.407+0000 7fda97dbbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.411 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.408+0000 7fda97dbbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.412 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.409+0000 7fda97dbbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:03.412 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.409+0000 7fda97dbbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:03.412 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.409+0000 7fda97dbbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:03.444 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:03.441+0000 7fda97dbbec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:48:06.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:48:06.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:06.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:48:06.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:06.795 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:48:07.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:48:07.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:07.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:07.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:07.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:07.298 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:48:07.322 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:07.320+0000 7f6dcb383ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.324 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:07.321+0000 7f6dcb383ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.326 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:07.324+0000 7f6dcb383ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:48:07.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:07.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:07.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:07.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:07.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:07.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:07.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:07.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:07.666 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:07.843 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:07.840+0000 7f6dcb383ec0 -1 Falling back to public interface 2020-03-11T22:48:08.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:08.719 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:08.717+0000 7f6dcb383ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:48:09.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:09.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:09.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:09.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:09.036 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:09.036 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:09.463 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63077,v1:127.0.0.1:6803/63077] [v2:127.0.0.1:6804/63077,v1:127.0.0.1:6805/63077] exists,up 43088468-da02-422f-86f4-3b7c265e3bc4 2020-03-11T22:48:09.463 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47 2020-03-11T22:48:09.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:09.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:09.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:09.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:48:09.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:09.466 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:09.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:09.468 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:09.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47 2020-03-11T22:48:09.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47' 2020-03-11T22:48:09.470 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:09.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCpamleeaxgHBAA0JpOH64PsaLRf3ud9aSo6w== 2020-03-11T22:48:09.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCpamleeaxgHBAA0JpOH64PsaLRf3ud9aSo6w=="}' 2020-03-11T22:48:09.479 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:48:09.969 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:09.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:48:09.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCpamleeaxgHBAA0JpOH64PsaLRf3ud9aSo6w== --osd-uuid ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47 2020-03-11T22:48:10.006 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.004+0000 7f6fdaa64ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:10.008 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.005+0000 7f6fdaa64ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:10.010 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.007+0000 7f6fdaa64ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:10.011 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.008+0000 7f6fdaa64ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:10.011 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.008+0000 7f6fdaa64ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:10.011 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.008+0000 7f6fdaa64ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:10.041 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:10.039+0000 7f6fdaa64ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:48:13.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:48:13.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:13.228 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:48:13.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:48:13.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:13.714 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:48:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:48:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:13.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:13.739 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:13.736+0000 7f7aa7528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:13.741 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:13.738+0000 7f7aa7528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:13.742 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:13.739+0000 7f7aa7528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:14.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:48:14.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:14.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:14.103 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:14.103 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:14.260 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:14.257+0000 7f7aa7528ec0 -1 Falling back to public interface 2020-03-11T22:48:14.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:14.949 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:14.946+0000 7f7aa7528ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:15.476 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:15.881 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63432,v1:127.0.0.1:6811/63432] [v2:127.0.0.1:6812/63432,v1:127.0.0.1:6813/63432] exists,up ef2fa8cf-d50d-4f81-9ea7-ee6f9aa8ba47 2020-03-11T22:48:15.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:15.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:15.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:15.885 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:15.886 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:15.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:48:15.888 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:15.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a7d9a769-1f20-4aef-a086-05e8ebe4d2a9 2020-03-11T22:48:15.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 a7d9a769-1f20-4aef-a086-05e8ebe4d2a9' 2020-03-11T22:48:15.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:15.890 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 a7d9a769-1f20-4aef-a086-05e8ebe4d2a9 2020-03-11T22:48:15.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCvamleMcNoNRAA+IQ9BpRarIJJDIFMyqNiMA== 2020-03-11T22:48:15.899 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCvamleMcNoNRAA+IQ9BpRarIJJDIFMyqNiMA=="}' 2020-03-11T22:48:15.899 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a7d9a769-1f20-4aef-a086-05e8ebe4d2a9 -i td/osd-backfill-prio/2/new.json 2020-03-11T22:48:16.484 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:48:16.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:48:16.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCvamleMcNoNRAA+IQ9BpRarIJJDIFMyqNiMA== --osd-uuid a7d9a769-1f20-4aef-a086-05e8ebe4d2a9 2020-03-11T22:48:16.522 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.519+0000 7ff289092ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:16.549 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.546+0000 7ff289092ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:16.549 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.547+0000 7ff289092ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:16.550 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.547+0000 7ff289092ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:16.550 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.547+0000 7ff289092ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:16.550 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.547+0000 7ff289092ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:16.607 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:16.605+0000 7ff289092ec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:48:19.689 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:48:19.690 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:19.691 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:48:19.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:48:19.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:20.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:48:20.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:20.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:20.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:20.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:20.179 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:48:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:20.199+0000 7f7f5128fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:20.201+0000 7f7f5128fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.206 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:20.202+0000 7f7f5128fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:48:20.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:20.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:20.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:20.613 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:20.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:20.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:20.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:20.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:20.614 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:20.723 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:20.720+0000 7f7f5128fec0 -1 Falling back to public interface 2020-03-11T22:48:20.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:21.499 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:21.495+0000 7f7f5128fec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:48:22.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:22.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:22.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:22.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:22.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:22.001 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:22.389 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63787,v1:127.0.0.1:6819/63787] [v2:127.0.0.1:6820/63787,v1:127.0.0.1:6821/63787] exists,up a7d9a769-1f20-4aef-a086-05e8ebe4d2a9 2020-03-11T22:48:22.389 INFO:tasks.workunit.client.0.smithi045.stdout:add osd3 ed2566e6-7354-4fdb-9848-bf1de3e4f211 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:48:22.390 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:22.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:22.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:48:22.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:22.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:48:22.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:48:22.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:22.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:22.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:22.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:22.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:22.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:22.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:22.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:22.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:48:22.396 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:22.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ed2566e6-7354-4fdb-9848-bf1de3e4f211 2020-03-11T22:48:22.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 ed2566e6-7354-4fdb-9848-bf1de3e4f211' 2020-03-11T22:48:22.396 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:22.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC2amlePGoAGBAATBoH6rvUEjkiGb+tw3AXoQ== 2020-03-11T22:48:22.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC2amlePGoAGBAATBoH6rvUEjkiGb+tw3AXoQ=="}' 2020-03-11T22:48:22.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ed2566e6-7354-4fdb-9848-bf1de3e4f211 -i td/osd-backfill-prio/3/new.json 2020-03-11T22:48:22.853 INFO:tasks.workunit.client.0.smithi045.stdout:3 2020-03-11T22:48:22.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:48:22.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC2amlePGoAGBAATBoH6rvUEjkiGb+tw3AXoQ== --osd-uuid ed2566e6-7354-4fdb-9848-bf1de3e4f211 2020-03-11T22:48:22.890 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.887+0000 7f34927a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:22.907 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.903+0000 7f34927a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:22.909 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.905+0000 7f34927a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:22.909 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.906+0000 7f34927a3ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:22.909 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.906+0000 7f34927a3ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:22.910 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.906+0000 7f34927a3ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:22.964 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:22.961+0000 7f34927a3ec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:48:26.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:48:26.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:26.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:48:26.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:26.145 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd3 key to auth repository 2020-03-11T22:48:26.651 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.3 2020-03-11T22:48:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:48:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:26.676 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:26.673+0000 7f5f7013eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:26.678 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:26.675+0000 7f5f7013eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:26.679 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:26.675+0000 7f5f7013eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:48:27.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:27.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:48:27.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:27.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:27.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:27.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:27.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:27.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:27.017 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:27.194 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:27.190+0000 7f5f7013eec0 -1 Falling back to public interface 2020-03-11T22:48:27.381 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:27.928 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:27.925+0000 7f5f7013eec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:48:28.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:28.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:28.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:28.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:28.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:28.383 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:28.830 INFO:tasks.workunit.client.0.smithi045.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64142,v1:127.0.0.1:6827/64142] [v2:127.0.0.1:6828/64142,v1:127.0.0.1:6829/64142] exists,up ed2566e6-7354-4fdb-9848-bf1de3e4f211 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stdout:add osd4 864cc8de-85a9-491a-bdbf-3858bb7818ff 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:28.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:28.832 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:28.833 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:48:28.834 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:28.835 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=864cc8de-85a9-491a-bdbf-3858bb7818ff 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 864cc8de-85a9-491a-bdbf-3858bb7818ff' 2020-03-11T22:48:28.836 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:28.847 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC8amleKENWMhAApHeaSc3fYtV3IbS5ozqHGA== 2020-03-11T22:48:28.847 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC8amleKENWMhAApHeaSc3fYtV3IbS5ozqHGA=="}' 2020-03-11T22:48:28.847 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 864cc8de-85a9-491a-bdbf-3858bb7818ff -i td/osd-backfill-prio/4/new.json 2020-03-11T22:48:29.385 INFO:tasks.workunit.client.0.smithi045.stdout:4 2020-03-11T22:48:29.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:48:29.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC8amleKENWMhAApHeaSc3fYtV3IbS5ozqHGA== --osd-uuid 864cc8de-85a9-491a-bdbf-3858bb7818ff 2020-03-11T22:48:29.422 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.418+0000 7fe6ed4b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:29.423 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.420+0000 7fe6ed4b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:29.425 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.422+0000 7fe6ed4b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:29.426 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.422+0000 7fe6ed4b7ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:29.426 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.422+0000 7fe6ed4b7ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:29.426 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.422+0000 7fe6ed4b7ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:29.523 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:29.520+0000 7fe6ed4b7ec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:48:32.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:48:32.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:32.726 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd4 key to auth repository 2020-03-11T22:48:32.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:48:32.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:48:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=5795ab0b-20d7-499c-b33e-bcd5c3558f5d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56115/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:33.230 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.4 2020-03-11T22:48:33.254 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:33.250+0000 7fc1a98b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.255 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:33.252+0000 7fc1a98b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.257 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:33.253+0000 7fc1a98b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:48:33.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:33.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:48:33.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:33.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:33.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:33.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:33.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:33.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:33.604 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:48:33.773 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:33.769+0000 7fc1a98b6ec0 -1 Falling back to public interface 2020-03-11T22:48:33.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:34.566 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:48:34.563+0000 7fc1a98b6ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:48:34.978 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:34.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:34.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:34.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:34.979 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:34.979 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:48:35.378 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:36.380 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:48:36.761 INFO:tasks.workunit.client.0.smithi045.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64497,v1:127.0.0.1:6835/64497] [v2:127.0.0.1:6836/64497,v1:127.0.0.1:6837/64497] exists,up 864cc8de-85a9-491a-bdbf-3858bb7818ff 2020-03-11T22:48:36.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:36.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:36.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:36.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:48:36.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:36.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:48:36.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:48:37.304 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test1' already exists 2020-03-11T22:48:37.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:38.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:48:39.190 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 1 size to 2 2020-03-11T22:48:39.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:39.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:48:39.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:48:39.728 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test2' already exists 2020-03-11T22:48:39.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:40.737 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:48:41.487 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 2 size to 2 2020-03-11T22:48:41.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:41.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:48:41.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:48:42.084 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test3' already exists 2020-03-11T22:48:42.092 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:43.093 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:48:43.725 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 3 size to 2 2020-03-11T22:48:43.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:43.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:48:43.733 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:48:44.316 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test4' already exists 2020-03-11T22:48:44.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:45.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:48:46.133 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 4 size to 2 2020-03-11T22:48:46.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:46.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:48:46.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:48:46.687 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test5' already exists 2020-03-11T22:48:46.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:47.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:48:48.355 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 5 size to 2 2020-03-11T22:48:48.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:48.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:48:48.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:48:49.071 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test6' already exists 2020-03-11T22:48:49.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:50.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:48:50.745 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 6 size to 2 2020-03-11T22:48:50.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:50.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:48:50.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:48:51.301 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test7' already exists 2020-03-11T22:48:51.309 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:52.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:48:53.011 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 7 size to 2 2020-03-11T22:48:53.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:53.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:48:53.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:48:53.558 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test8' already exists 2020-03-11T22:48:53.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:54.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:48:55.336 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 8 size to 2 2020-03-11T22:48:55.346 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:55.346 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:48:55.346 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:48:56.021 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test9' already exists 2020-03-11T22:48:56.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:57.031 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:48:57.816 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 9 size to 2 2020-03-11T22:48:57.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:57.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:48:57.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:48:58.404 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test10' already exists 2020-03-11T22:48:58.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:59.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:49:00.130 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 10 size to 2 2020-03-11T22:49:00.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:49:05.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:05.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:05.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:05.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:05.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:05.141 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:05.141 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:05.141 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:05.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:05.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:05.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:05.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:05.248 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:05.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:05.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:05.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:49:05.631 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:05.632 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:05.632 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:05.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-03-11T22:49:05.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495' 2020-03-11T22:49:05.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:05.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:05.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:49:05.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-38654705676' 2020-03-11T22:49:05.895 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:05.895 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:06.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:49:06.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-38654705676 2-60129542155' 2020-03-11T22:49:06.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:06.028 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:06.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378634 2020-03-11T22:49:06.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-38654705676 2-60129542155 3-81604378634' 2020-03-11T22:49:06.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:06.166 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:06.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215113 2020-03-11T22:49:06.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-38654705676 2-60129542155 3-81604378634 4-103079215113' 2020-03-11T22:49:06.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:06.293 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836495 2020-03-11T22:49:06.293 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:06.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:06.294 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836495 2020-03-11T22:49:06.294 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:06.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836495 2020-03-11T22:49:06.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-03-11T22:49:06.295 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836495 2020-03-11T22:49:06.295 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:06.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836495 -lt 21474836495 2020-03-11T22:49:06.665 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:06.665 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:49:06.665 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:06.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:06.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:49:06.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:06.667 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:49:07.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705677 -lt 38654705676 2020-03-11T22:49:07.037 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.038 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:49:07.038 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:07.039 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:49:07.039 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:49:07.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:49:07.040 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:49:07.041 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:07.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:49:07.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.432 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378634 2020-03-11T22:49:07.432 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:07.433 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378634 2020-03-11T22:49:07.433 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378634 2020-03-11T22:49:07.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2020-03-11T22:49:07.435 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:07.435 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378634 2020-03-11T22:49:07.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378634 -lt 81604378634 2020-03-11T22:49:07.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.801 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215113 2020-03-11T22:49:07.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:07.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215113 2020-03-11T22:49:07.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215113 2020-03-11T22:49:07.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215113' 2020-03-11T22:49:07.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:07.804 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215113 2020-03-11T22:49:08.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215113 -lt 103079215113 2020-03-11T22:49:08.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:08.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:08.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:08.663 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:49:08.663 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:08.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:08.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:08.664 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:08.664 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:08.664 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:08.664 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:09.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:49:09.020 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:09.021 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:09.021 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:09.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:49:09.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:09.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:09.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:09.851 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:09.851 INFO:tasks.workunit.client.0.smithi045.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:00.289287+0000 0'0 81:19 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:48:58.149845+0000 0'0 2020-03-11T22:48:58.149845+0000 0 2020-03-11T22:49:09.851 INFO:tasks.workunit.client.0.smithi045.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:58.421376+0000 0'0 80:24 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:55.813513+0000 0'0 2020-03-11T22:48:55.813513+0000 0 2020-03-11T22:49:09.851 INFO:tasks.workunit.client.0.smithi045.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:55.538409+0000 0'0 78:19 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:53.350223+0000 0'0 2020-03-11T22:48:53.350223+0000 0 2020-03-11T22:49:09.851 INFO:tasks.workunit.client.0.smithi045.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:53.628176+0000 0'0 80:33 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:51.074198+0000 0'0 2020-03-11T22:48:51.074198+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:41.576718+0000 0'0 78:46 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:48:39.518813+0000 0'0 2020-03-11T22:48:39.518813+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:39.319510+0000 0'0 80:53 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:37.115158+0000 0'0 2020-03-11T22:48:37.115158+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:44.919187+0000 0'0 80:39 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:41.841266+0000 0'0 2020-03-11T22:48:41.841266+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:46.889361+0000 0'0 79:42 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:48:44.070015+0000 0'0 2020-03-11T22:48:44.070015+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:48.770085+0000 0'0 78:35 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:46.464186+0000 0'0 2020-03-11T22:48:46.464186+0000 0 2020-03-11T22:49:09.852 INFO:tasks.workunit.client.0.smithi045.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:51.953453+0000 0'0 79:33 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:48:48.826902+0000 0'0 2020-03-11T22:48:48.826902+0000 0 2020-03-11T22:49:09.853 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:49:09.853 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:09.853 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:49:09.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:49:09.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:49:09.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:49:09.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:49:09.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:49:09.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:49:09.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:09.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:09.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:49:09.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:10.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:10.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:10.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:10.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:49:10.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:10.593 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:10.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:49:10.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:10.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:49:10.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:49:10.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:10.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:49:10.596 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:10.955 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:10.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:49:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:49:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:49:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:49:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:49:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:49:10.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:11.314 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:11.315 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:11.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:11.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:49:11.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:49:11.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:11.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:49:11.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:11.708 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.708 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:11.709 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:11.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:11.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:49:11.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:11.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:11.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:49:11.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:12.136 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.137 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:12.137 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:12.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:12.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:49:12.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:49:12.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:49:12.139 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:12.514 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.515 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:12.515 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:12.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:12.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:49:12.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:12.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:49:12.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:49:12.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:49:12.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:49:12.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:49:12.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:49:12.521 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:12.522 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:49:12.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:49:13.073 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test2' does not exist 2020-03-11T22:49:13.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:49:13.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:49:13.650 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test4' does not exist 2020-03-11T22:49:13.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:13.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:49:13.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:49:13.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:49:13.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:49:14.231 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test5' does not exist 2020-03-11T22:49:14.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:49:14.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:49:14.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:49:14.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:49:14.817 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test6' does not exist 2020-03-11T22:49:14.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:49:14.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:49:14.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:49:14.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:49:14.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:49:15.399 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test8' does not exist 2020-03-11T22:49:15.407 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:15.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:49:15.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:49:15.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:49:15.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:49:15.950 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test9' does not exist 2020-03-11T22:49:15.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:15.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:49:15.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:49:15.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:49:15.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:49:16.490 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test10' does not exist 2020-03-11T22:49:16.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:49:17.244 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 3 size to 1 2020-03-11T22:49:17.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:49:17.987 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 7 size to 1 2020-03-11T22:49:17.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:17.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:17.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:17.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:17.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:17.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:17.997 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:17.997 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:17.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:17.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:17.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:18.091 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:18.091 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:18.092 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:18.092 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:18.092 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:18.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:18.502 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:49:18.502 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:49:18.502 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:49:18.503 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:49:18.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:18.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.503 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:18.636 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836499 2020-03-11T22:49:18.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499' 2020-03-11T22:49:18.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:18.774 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705681 2020-03-11T22:49:18.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-38654705681' 2020-03-11T22:49:18.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.775 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:18.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542160 2020-03-11T22:49:18.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-38654705681 2-60129542160' 2020-03-11T22:49:18.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.906 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:19.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378638 2020-03-11T22:49:19.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-38654705681 2-60129542160 3-81604378638' 2020-03-11T22:49:19.038 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:19.039 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:19.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215117 2020-03-11T22:49:19.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-38654705681 2-60129542160 3-81604378638 4-103079215117' 2020-03-11T22:49:19.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:19.171 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836499 2020-03-11T22:49:19.171 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:19.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:19.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836499 2020-03-11T22:49:19.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:19.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836499 2020-03-11T22:49:19.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-03-11T22:49:19.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:19.174 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836499 2020-03-11T22:49:19.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836499 2020-03-11T22:49:19.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:20.539 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:20.539 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:20.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836500 -lt 21474836499 2020-03-11T22:49:20.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:20.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705681 2020-03-11T22:49:20.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:20.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:20.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705681 2020-03-11T22:49:20.926 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:20.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705681 2020-03-11T22:49:20.927 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2020-03-11T22:49:20.927 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:20.927 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705681 2020-03-11T22:49:21.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705681 -lt 38654705681 2020-03-11T22:49:21.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542160 2020-03-11T22:49:21.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:21.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542160 2020-03-11T22:49:21.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.302 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542160 2020-03-11T22:49:21.303 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2020-03-11T22:49:21.303 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:21.303 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542160 2020-03-11T22:49:21.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542160 -lt 60129542160 2020-03-11T22:49:21.693 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378638 2020-03-11T22:49:21.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:21.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378638 2020-03-11T22:49:21.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378638 2020-03-11T22:49:21.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-03-11T22:49:21.696 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:21.696 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378638 2020-03-11T22:49:22.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378639 -lt 81604378638 2020-03-11T22:49:22.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:22.065 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215117 2020-03-11T22:49:22.065 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:22.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:22.067 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215117 2020-03-11T22:49:22.067 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:22.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215117 2020-03-11T22:49:22.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215117' 2020-03-11T22:49:22.068 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:22.068 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215117 2020-03-11T22:49:22.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215117 -lt 103079215117 2020-03-11T22:49:22.452 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:22.452 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:22.453 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:22.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:49:22.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:22.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:22.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:22.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:22.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:22.939 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:22.939 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:23.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:49:23.297 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:23.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:23.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:49:23.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:23.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:23.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:49:23.842 INFO:tasks.workunit.client.0.smithi045.stderr:10+0 records in 2020-03-11T22:49:23.842 INFO:tasks.workunit.client.0.smithi045.stderr:10+0 records out 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0459487 s, 228 MB/s 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:23.843 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:49:24.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:24.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:49:24.821 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:24.822 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:49:25.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:25.285 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:49:25.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:25.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:49:26.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:26.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:49:26.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:26.829 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:49:27.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:27.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:49:27.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:27.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.327 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.711 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.317 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.254 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.254 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.738 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:49:37.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:37.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.020 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.454 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.454 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.463 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.307 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.307 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.388 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.388 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:49:45.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.434 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:49:45.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:49:46.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.528 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:49:47.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:49:47.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:49:47.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:49:48.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.467 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:49:48.973 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:49:48.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:49:48.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:48.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:48.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.235 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.534 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.535 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.819 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.819 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.111 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.369 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.011 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.419 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.245 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.245 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.558 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.558 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.899 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.029 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.828 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.828 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.386 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.387 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.251 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.251 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.578 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.579 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.837 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.837 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.145 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.146 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.404 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.545 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.545 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.822 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.822 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.522 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:50:03.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:50:03.523 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:50:03.524 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:50:03.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:50:03.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.723 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.599 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.301 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.633 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.126 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.685 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.210 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.352 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.985 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:50:18.236 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:50:18.236 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:50:18.237 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:50:18.237 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:50:18.238 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:50:18.238 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:50:18.238 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:50:18.238 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:50:18.239 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:18.239 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:50:18.626 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:50:18.627 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:18.627 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:50:19.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:50:19.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:19.372 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:19.373 INFO:tasks.workunit.client.0.smithi045.stdout:7.0 43 0 0 0 0 450887680 0 0 29 29 active+clean 2020-03-11T22:49:18.351784+0000 93'129 93:176 [1] 1 [1] 1 0'0 2020-03-11T22:48:51.074198+0000 0'0 2020-03-11T22:48:51.074198+0000 0 2020-03-11T22:50:19.373 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:39.319510+0000 93'150 93:210 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:37.115158+0000 0'0 2020-03-11T22:48:37.115158+0000 0 2020-03-11T22:50:19.373 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:18.360079+0000 93'150 93:203 [1] 1 [1] 1 0'0 2020-03-11T22:48:41.841266+0000 0'0 2020-03-11T22:48:41.841266+0000 0 2020-03-11T22:50:19.373 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:50:19.373 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:19.374 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:50:19.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:50:19.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:50:20.113 INFO:tasks.workunit.client.0.smithi045.stderr:nobackfill is set 2020-03-11T22:50:20.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:50:20.864 INFO:tasks.workunit.client.0.smithi045.stderr:noout is set 2020-03-11T22:50:20.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:50:21.626 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 7 size to 2 2020-03-11T22:50:21.634 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:50:23.636 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:23.637 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:23.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:23.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:23.740 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:23.741 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:23.742 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:23.742 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:23.742 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:23.742 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:23.742 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:23.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:23.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:23.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:23.749 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:24.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:50:24.106 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:50:26.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:26.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:26.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:26.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:50:26.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:26.481 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:26.481 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:26.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.860 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:26.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836515 2020-03-11T22:50:26.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515' 2020-03-11T22:50:26.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:27.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705697 2020-03-11T22:50:27.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705697' 2020-03-11T22:50:27.125 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:27.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:27.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542175 2020-03-11T22:50:27.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705697 2-60129542175' 2020-03-11T22:50:27.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:27.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:27.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378654 2020-03-11T22:50:27.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705697 2-60129542175 3-81604378654' 2020-03-11T22:50:27.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:27.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:27.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215133 2020-03-11T22:50:27.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705697 2-60129542175 3-81604378654 4-103079215133' 2020-03-11T22:50:27.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:27.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836515 2020-03-11T22:50:27.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:27.525 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:27.526 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836515 2020-03-11T22:50:27.526 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:27.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836515 2020-03-11T22:50:27.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2020-03-11T22:50:27.527 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:27.527 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836515 2020-03-11T22:50:27.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836513 -lt 21474836515 2020-03-11T22:50:27.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:28.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:28.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:29.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836515 2020-03-11T22:50:29.274 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:29.274 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705697 2020-03-11T22:50:29.274 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:29.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:29.275 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705697 2020-03-11T22:50:29.276 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:29.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705697 2020-03-11T22:50:29.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2020-03-11T22:50:29.277 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705697 2020-03-11T22:50:29.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:29.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705697 -lt 38654705697 2020-03-11T22:50:29.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:29.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542175 2020-03-11T22:50:29.643 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:29.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:29.644 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542175 2020-03-11T22:50:29.644 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:29.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542175 2020-03-11T22:50:29.645 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542175' 2020-03-11T22:50:29.645 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542175 2020-03-11T22:50:29.645 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:30.079 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542176 -lt 60129542175 2020-03-11T22:50:30.080 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:30.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378654 2020-03-11T22:50:30.080 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:30.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:30.081 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378654 2020-03-11T22:50:30.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:30.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378654 2020-03-11T22:50:30.083 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378654' 2020-03-11T22:50:30.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:30.083 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378654 2020-03-11T22:50:30.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378654 -lt 81604378654 2020-03-11T22:50:30.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:30.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215133 2020-03-11T22:50:30.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:30.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:30.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215133 2020-03-11T22:50:30.479 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:30.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215133 2020-03-11T22:50:30.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215133' 2020-03-11T22:50:30.480 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215133 2020-03-11T22:50:30.480 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:30.848 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215133 -lt 103079215133 2020-03-11T22:50:30.849 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:30.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:30.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:30.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:30.942 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:30.943 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:30.945 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:30.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:50:31.544 INFO:tasks.workunit.client.0.smithi045.stderr:osd.0 is already out. 2020-03-11T22:50:31.552 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:50:33.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:33.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:33.554 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:33.932 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:33.932 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:50:33.932 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:50:33.932 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:50:33.932 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:50:33.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:33.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:33.933 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:34.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836518 2020-03-11T22:50:34.073 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518' 2020-03-11T22:50:34.074 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:34.074 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:34.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705700 2020-03-11T22:50:34.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-38654705700' 2020-03-11T22:50:34.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:34.206 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:34.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542179 2020-03-11T22:50:34.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-38654705700 2-60129542179' 2020-03-11T22:50:34.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:34.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:34.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378658 2020-03-11T22:50:34.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-38654705700 2-60129542179 3-81604378658' 2020-03-11T22:50:34.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:34.472 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:34.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215136 2020-03-11T22:50:34.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-38654705700 2-60129542179 3-81604378658 4-103079215136' 2020-03-11T22:50:34.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:34.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836518 2020-03-11T22:50:34.606 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:34.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:34.606 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836518 2020-03-11T22:50:34.606 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:34.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836518 2020-03-11T22:50:34.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2020-03-11T22:50:34.608 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:34.608 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836518 2020-03-11T22:50:34.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836518 -lt 21474836518 2020-03-11T22:50:34.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:34.981 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705700 2020-03-11T22:50:34.981 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:34.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:34.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705700 2020-03-11T22:50:34.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:34.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705700 2020-03-11T22:50:34.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2020-03-11T22:50:34.984 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:34.984 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705700 2020-03-11T22:50:35.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705700 -lt 38654705700 2020-03-11T22:50:35.391 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.391 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542179 2020-03-11T22:50:35.392 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.392 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:35.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542179 2020-03-11T22:50:35.393 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.393 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542179 2020-03-11T22:50:35.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542179' 2020-03-11T22:50:35.394 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542179 2020-03-11T22:50:35.394 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:35.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542179 -lt 60129542179 2020-03-11T22:50:35.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378658 2020-03-11T22:50:35.783 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:35.784 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378658 2020-03-11T22:50:35.785 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378658 2020-03-11T22:50:35.786 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378658' 2020-03-11T22:50:35.786 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:35.786 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378658 2020-03-11T22:50:36.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378656 -lt 81604378658 2020-03-11T22:50:36.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:37.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:37.195 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:37.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378658 -lt 81604378658 2020-03-11T22:50:37.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:37.585 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215136 2020-03-11T22:50:37.585 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:37.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:37.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215136 2020-03-11T22:50:37.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:37.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215136 2020-03-11T22:50:37.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215136' 2020-03-11T22:50:37.588 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215136 2020-03-11T22:50:37.588 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:37.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215137 -lt 103079215136 2020-03-11T22:50:37.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:37.975 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:37.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:37.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:38.069 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:38.070 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:38.071 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:38.073 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:38.074 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:38.074 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:38.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:38.538 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:38.538 INFO:tasks.workunit.client.0.smithi045.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:26.468056+0000 93'150 103:215 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:51.074198+0000 0'0 2020-03-11T22:48:51.074198+0000 0 2020-03-11T22:50:38.539 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:32.607991+0000 93'150 104:226 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:37.115158+0000 0'0 2020-03-11T22:48:37.115158+0000 0 2020-03-11T22:50:38.539 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:18.360079+0000 93'150 103:211 [1] 1 [1] 1 0'0 2020-03-11T22:48:41.841266+0000 0'0 2020-03-11T22:48:41.841266+0000 0 2020-03-11T22:50:38.539 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:50:38.539 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:38.540 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:50:38.548 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:50:39.161 INFO:tasks.workunit.client.0.smithi045.stderr:set pool 3 size to 2 2020-03-11T22:50:39.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:50:41.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:41.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:41.172 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:41.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:41.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:41.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836522 2020-03-11T22:50:41.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522' 2020-03-11T22:50:41.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:41.754 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:41.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705704 2020-03-11T22:50:41.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705704' 2020-03-11T22:50:41.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:41.898 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542182 2020-03-11T22:50:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542182' 2020-03-11T22:50:42.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:42.030 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:42.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378661 2020-03-11T22:50:42.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542182 3-81604378661' 2020-03-11T22:50:42.163 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:42.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:42.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215140 2020-03-11T22:50:42.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542182 3-81604378661 4-103079215140' 2020-03-11T22:50:42.296 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.296 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836522 2020-03-11T22:50:42.297 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.297 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:42.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836522 2020-03-11T22:50:42.298 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836522 2020-03-11T22:50:42.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2020-03-11T22:50:42.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:42.299 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836522 2020-03-11T22:50:42.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836522 -lt 21474836522 2020-03-11T22:50:42.696 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.697 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705704 2020-03-11T22:50:42.697 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:42.697 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705704 2020-03-11T22:50:42.698 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705704 2020-03-11T22:50:42.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705704' 2020-03-11T22:50:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:42.699 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705704 2020-03-11T22:50:43.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705704 -lt 38654705704 2020-03-11T22:50:43.065 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:43.065 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542182 2020-03-11T22:50:43.066 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:43.066 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:43.066 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542182 2020-03-11T22:50:43.067 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:43.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542182 2020-03-11T22:50:43.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542182' 2020-03-11T22:50:43.068 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:43.068 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542182 2020-03-11T22:50:43.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542182 -lt 60129542182 2020-03-11T22:50:43.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:43.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378661 2020-03-11T22:50:43.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:43.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:43.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378661 2020-03-11T22:50:43.438 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:43.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378661 2020-03-11T22:50:43.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378661' 2020-03-11T22:50:43.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:43.439 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378661 2020-03-11T22:50:43.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378661 -lt 81604378661 2020-03-11T22:50:43.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:43.891 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215140 2020-03-11T22:50:43.891 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:43.891 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:43.892 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215140 2020-03-11T22:50:43.892 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:43.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215140 2020-03-11T22:50:43.893 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215140' 2020-03-11T22:50:43.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:43.893 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215140 2020-03-11T22:50:44.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215140 -lt 103079215140 2020-03-11T22:50:44.265 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:44.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:44.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:44.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:44.366 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:44.367 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:50:44.368 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:44.369 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:44.371 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:44.371 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:44.371 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:44.371 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:44.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:44.709 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:44.710 INFO:tasks.workunit.client.0.smithi045.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:26.468056+0000 93'150 107:218 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:51.074198+0000 0'0 2020-03-11T22:48:51.074198+0000 0 2020-03-11T22:50:44.710 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:32.607991+0000 93'150 107:229 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:37.115158+0000 0'0 2020-03-11T22:48:37.115158+0000 0 2020-03-11T22:50:44.710 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 activating+undersized+remapped 2020-03-11T22:50:41.876129+0000 93'150 108:217 [1,2] 1 [1] 1 0'0 2020-03-11T22:48:41.841266+0000 0'0 2020-03-11T22:48:41.841266+0000 0 2020-03-11T22:50:44.710 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:50:44.710 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:44.711 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:50:44.720 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:44.720 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:50:44.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:50:44.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:50:44.723 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:44.723 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:44.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:50:44.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:50:44.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:50:44.726 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:44.726 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:44.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:44.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:44.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:44.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:44.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:50:44.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:45.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:218: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:50:45.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:222: TEST_backfill_priority: sleep 2 2020-03-11T22:50:47.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:47.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:50:47.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:47.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:50:47.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:50:49.464 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:49.464 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:49.464 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:49.464 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:49.464 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:49.465 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:49.465 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:49.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:49.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:49.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:49.576 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:49.576 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:49.576 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:49.578 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 254, 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:49.580 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:49.581 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:49.582 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:49.582 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:49.583 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:49.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:49.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:50.082 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836526 2020-03-11T22:50:50.083 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526' 2020-03-11T22:50:50.083 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:50.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:50.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705707 2020-03-11T22:50:50.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705707' 2020-03-11T22:50:50.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:50.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:50.348 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542186 2020-03-11T22:50:50.349 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705707 2-60129542186' 2020-03-11T22:50:50.349 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:50.349 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:50.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378665 2020-03-11T22:50:50.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705707 2-60129542186 3-81604378665' 2020-03-11T22:50:50.482 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:50.482 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:50.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215143 2020-03-11T22:50:50.615 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705707 2-60129542186 3-81604378665 4-103079215143' 2020-03-11T22:50:50.615 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:50.615 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836526 2020-03-11T22:50:50.615 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:50.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:50.616 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836526 2020-03-11T22:50:50.616 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:50.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836526 2020-03-11T22:50:50.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2020-03-11T22:50:50.617 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:50.618 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836526 2020-03-11T22:50:50.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836526 -lt 21474836526 2020-03-11T22:50:50.986 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:50.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705707 2020-03-11T22:50:50.987 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:50.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:50.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705707 2020-03-11T22:50:50.988 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:50.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705707 2020-03-11T22:50:50.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705707' 2020-03-11T22:50:50.989 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:50.989 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705707 2020-03-11T22:50:51.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705707 -lt 38654705707 2020-03-11T22:50:51.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:51.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542186 2020-03-11T22:50:51.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:51.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:51.360 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542186 2020-03-11T22:50:51.360 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:51.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542186 2020-03-11T22:50:51.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542186' 2020-03-11T22:50:51.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:51.362 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542186 2020-03-11T22:50:51.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542186 -lt 60129542186 2020-03-11T22:50:51.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:51.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378665 2020-03-11T22:50:51.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:51.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:51.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378665 2020-03-11T22:50:51.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:51.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378665 2020-03-11T22:50:51.766 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378665' 2020-03-11T22:50:51.766 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:51.766 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378665 2020-03-11T22:50:52.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378663 -lt 81604378665 2020-03-11T22:50:52.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:53.191 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:53.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:53.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378665 -lt 81604378665 2020-03-11T22:50:53.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:53.569 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215143 2020-03-11T22:50:53.569 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:53.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:53.570 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215143 2020-03-11T22:50:53.570 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:53.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215143 2020-03-11T22:50:53.571 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215143' 2020-03-11T22:50:53.571 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215143 2020-03-11T22:50:53.572 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:53.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215144 -lt 103079215143 2020-03-11T22:50:53.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:50:54.296 INFO:tasks.workunit.client.0.smithi045.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:50:54.296 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:50:54.304 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:50:56.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:56.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:56.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:56.307 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:56.307 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:56.307 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:50:56.307 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:50:56.308 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:56.308 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:56.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:56.419 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:50:56.419 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:50:56.419 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:56.419 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:50:56.420 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:50:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:56.422 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:56.423 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:50:56.424 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:50:56.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:56.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:50:56.426 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:50:56.426 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:50:56.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:56.427 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:56.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:56.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:56.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:50:56.784 INFO:tasks.workunit.client.0.smithi045.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:50:56.784 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:50:56.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:51:01.795 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:51:01.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:01.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:01.898 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:01.898 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:01.898 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:01.898 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:01.898 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:01.899 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:01.900 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:01.901 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:01.902 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:01.903 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:01.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:01.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:01.906 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:02.288 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:02.289 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:51:02.289 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:51:02.289 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:51:02.289 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:51:02.289 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:02.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:02.290 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:02.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836530 2020-03-11T22:51:02.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530' 2020-03-11T22:51:02.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:02.423 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:02.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705712 2020-03-11T22:51:02.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705712' 2020-03-11T22:51:02.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:02.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:02.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542190 2020-03-11T22:51:02.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705712 2-60129542190' 2020-03-11T22:51:02.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:02.730 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:02.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378669 2020-03-11T22:51:02.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705712 2-60129542190 3-81604378669' 2020-03-11T22:51:02.863 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:02.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215148 2020-03-11T22:51:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705712 2-60129542190 3-81604378669 4-103079215148' 2020-03-11T22:51:02.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:02.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836530 2020-03-11T22:51:02.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:02.997 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:02.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836530 2020-03-11T22:51:02.997 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:02.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836530 2020-03-11T22:51:02.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2020-03-11T22:51:02.999 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:02.999 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836530 2020-03-11T22:51:03.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836530 -lt 21474836530 2020-03-11T22:51:03.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:03.365 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705712 2020-03-11T22:51:03.365 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:03.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:03.366 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705712 2020-03-11T22:51:03.366 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:03.367 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705712 2020-03-11T22:51:03.367 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705712' 2020-03-11T22:51:03.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:03.368 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705712 2020-03-11T22:51:03.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705710 -lt 38654705712 2020-03-11T22:51:03.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:04.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:04.797 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:05.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705712 -lt 38654705712 2020-03-11T22:51:05.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:05.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542190 2020-03-11T22:51:05.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:05.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:05.188 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542190 2020-03-11T22:51:05.188 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:05.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542190 2020-03-11T22:51:05.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542190' 2020-03-11T22:51:05.189 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:05.189 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542190 2020-03-11T22:51:05.555 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542191 -lt 60129542190 2020-03-11T22:51:05.556 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:05.556 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378669 2020-03-11T22:51:05.556 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:05.557 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:05.558 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378669 2020-03-11T22:51:05.558 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:05.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378669 2020-03-11T22:51:05.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378669' 2020-03-11T22:51:05.559 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:05.559 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378669 2020-03-11T22:51:05.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378670 -lt 81604378669 2020-03-11T22:51:05.928 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:05.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215148 2020-03-11T22:51:05.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:05.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:05.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215148 2020-03-11T22:51:05.930 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:05.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215148 2020-03-11T22:51:05.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215148' 2020-03-11T22:51:05.931 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:05.931 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215148 2020-03-11T22:51:06.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215148 -lt 103079215148 2020-03-11T22:51:06.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:06.665 INFO:tasks.workunit.client.0.smithi045.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:51:06.666 INFO:tasks.workunit.client.0.smithi045.stderr: 2020-03-11T22:51:06.673 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:08.676 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:51:08.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:51:08.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:08.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:08.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:08.795 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:08.795 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:08.795 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:08.795 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:08.796 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:08.797 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:08.798 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:08.799 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:08.800 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:08.801 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:08.801 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:08.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:08.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:51:08.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:51:08.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:08.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:08.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:08.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:08.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:51:09.472 INFO:tasks.workunit.client.0.smithi045.stderr:noout is unset 2020-03-11T22:51:09.480 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:51:10.141 INFO:tasks.workunit.client.0.smithi045.stderr:nobackfill is unset 2020-03-11T22:51:10.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:10.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:10.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:51:10.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:51:10.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:51:10.152 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:51:10.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:51:10.153 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:51:10.153 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:51:10.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:51:10.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:51:10.153 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:51:10.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:51:10.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:51:10.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:51:10.246 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:10.246 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:10.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:4' 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:10.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:10.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:10.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836534 2020-03-11T22:51:10.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836534' 2020-03-11T22:51:10.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:10.825 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:10.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705715 2020-03-11T22:51:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836534 1-38654705715' 2020-03-11T22:51:10.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:10.958 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:11.090 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542194 2020-03-11T22:51:11.091 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836534 1-38654705715 2-60129542194' 2020-03-11T22:51:11.091 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:11.091 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:11.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378673 2020-03-11T22:51:11.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836534 1-38654705715 2-60129542194 3-81604378673' 2020-03-11T22:51:11.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:11.225 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:11.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215152 2020-03-11T22:51:11.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836534 1-38654705715 2-60129542194 3-81604378673 4-103079215152' 2020-03-11T22:51:11.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:11.358 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836534 2020-03-11T22:51:11.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:11.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:11.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836534 2020-03-11T22:51:11.360 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:11.360 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836534 2020-03-11T22:51:11.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2020-03-11T22:51:11.361 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:11.361 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836534 2020-03-11T22:51:11.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836532 -lt 21474836534 2020-03-11T22:51:11.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:12.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:12.730 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:13.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836534 -lt 21474836534 2020-03-11T22:51:13.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:13.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705715 2020-03-11T22:51:13.114 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:13.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:13.115 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705715 2020-03-11T22:51:13.115 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:13.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705715 2020-03-11T22:51:13.116 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705715' 2020-03-11T22:51:13.116 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:13.116 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705715 2020-03-11T22:51:13.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705716 -lt 38654705715 2020-03-11T22:51:13.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:13.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542194 2020-03-11T22:51:13.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:13.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:13.485 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542194 2020-03-11T22:51:13.485 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:13.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542194 2020-03-11T22:51:13.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542194' 2020-03-11T22:51:13.486 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:13.486 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542194 2020-03-11T22:51:13.922 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542194 -lt 60129542194 2020-03-11T22:51:13.922 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:13.923 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378673 2020-03-11T22:51:13.923 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:13.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:13.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378673 2020-03-11T22:51:13.924 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378673 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378673' 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378673 2020-03-11T22:51:14.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378673 -lt 81604378673 2020-03-11T22:51:14.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:14.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215152 2020-03-11T22:51:14.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:14.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:14.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215152 2020-03-11T22:51:14.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:14.301 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215152 2020-03-11T22:51:14.301 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215152' 2020-03-11T22:51:14.302 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:14.302 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215152 2020-03-11T22:51:14.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215152 -lt 103079215152 2020-03-11T22:51:14.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:51:14.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:14.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:15.249 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:15.250 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:15.250 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:15.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:15.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:15.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:15.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:16.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:16.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:51:16.086 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:16.178 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:16.178 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:16.178 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:16.178 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:16.179 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:16.180 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:16.181 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:16.182 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:16.183 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:16.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:16.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:16.287 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:16.288 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:16.288 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:16.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:16.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:16.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:16.289 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:16.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:16.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:16.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:16.650 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:17.123 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:17.124 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:17.125 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:17.604 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19578403 2020-03-11T22:51:17.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19578403 '!=' null 2020-03-11T22:51:17.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:17.605 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:17.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:17.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:17.697 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:17.699 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:17.700 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:17.701 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:17.704 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:17.705 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:17.807 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:17.808 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:17.808 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:18.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:18.167 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:18.167 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:18.167 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:18.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:18.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:18.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:18.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:18.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:18.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:18.677 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:18.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:18.677 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:19.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15968303 2020-03-11T22:51:19.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15968303 '!=' null 2020-03-11T22:51:19.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:19.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:19.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:19.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:19.252 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:19.252 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:19.253 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:19.255 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:19.256 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:19.257 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:19.258 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:19.258 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:19.258 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:19.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:19.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:19.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:19.362 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:19.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:19.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:19.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:19.728 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:20.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:20.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:20.204 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:20.204 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13241144 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13241144 '!=' null 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:20.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:20.946 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:20.947 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:20.948 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:20.948 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:20.948 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:20.948 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:20.948 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:20.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:21.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:21.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:21.053 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:21.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:21.410 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:21.411 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:21.899 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:21.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:21.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:21.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:21.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:22.400 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13241144 2020-03-11T22:51:22.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13241144 '!=' null 2020-03-11T22:51:22.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:22.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:22.401 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:22.401 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:22.494 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:22.494 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:22.494 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:22.495 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:22.496 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:22.497 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:22.498 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:22.500 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:22.500 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:22.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:22.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:22.603 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:22.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:22.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:22.603 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:22.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:22.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:22.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:22.961 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:22.961 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:22.962 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:22.962 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:23.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:23.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:23.449 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:23.450 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841679 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841679 '!=' null 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:23.959 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:24.051 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:24.051 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:24.052 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:24.053 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:24.054 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:24.056 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:24.057 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:24.057 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:24.057 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:24.058 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:24.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:24.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:24.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:24.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:24.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:24.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:24.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:24.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:24.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:24.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:24.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:24.563 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:25.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:25.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:25.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:25.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:25.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:25.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:25.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:25.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:25.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:25.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967928 2020-03-11T22:51:25.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967928 '!=' null 2020-03-11T22:51:25.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:25.524 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:25.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:25.525 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:25.616 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:25.616 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:25.616 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:25.616 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:25.616 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:25.617 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:25.617 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:25.617 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:25.617 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:25.618 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:25.619 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:25.620 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:25.621 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:25.622 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:25.622 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:25.622 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:25.622 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:25.622 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:25.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:25.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:25.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:25.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:25.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:25.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:25.726 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:25.726 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:25.726 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:26.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:26.088 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:26.088 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:26.088 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:26.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:26.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:26.568 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:27.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578420 2020-03-11T22:51:27.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578420 '!=' null 2020-03-11T22:51:27.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:27.039 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.039 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:27.040 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:27.137 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:27.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:27.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:27.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:27.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:27.240 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:27.241 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:27.241 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:27.598 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:27.598 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:27.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:27.599 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:28.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:28.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:28.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:28.087 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:28.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:28.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:28.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:28.088 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:28.088 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:28.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578420 2020-03-11T22:51:28.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578420 '!=' null 2020-03-11T22:51:28.663 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:28.663 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:28.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:28.663 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:28.755 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:28.755 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:28.755 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:28.755 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:28.756 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:28.756 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:28.756 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:28.756 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:28.757 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:28.757 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:28.757 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:28.757 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:28.758 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:28.758 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:28.758 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:28.758 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:28.758 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:28.759 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:28.759 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:28.759 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:28.759 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:28.760 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:28.760 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:28.760 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:28.760 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:28.760 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:28.761 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:28.762 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:28.763 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:28.763 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:28.864 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:28.865 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:28.865 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:29.217 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:29.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:29.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:29.218 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:29.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:29.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:29.702 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094374 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094374 '!=' null 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:30.176 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.268 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:30.268 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:30.268 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:30.268 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:30.268 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:30.269 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:30.270 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:30.271 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:30.272 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:30.273 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:30.276 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:30.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:30.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:30.381 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:30.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:30.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:30.792 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:30.793 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:31.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:31.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:31.280 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:31.281 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:31.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094365 2020-03-11T22:51:31.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094365 '!=' null 2020-03-11T22:51:31.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:31.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:31.805 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:31.805 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:31.896 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:31.896 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:31.897 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:31.898 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 151, 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:31.899 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "7.0", 2020-03-11T22:51:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 254, 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:31.901 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:31.902 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:31.902 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:31.902 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:31.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:32.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:32.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:32.006 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:32.006 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:32.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:32.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:32.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:32.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:32.367 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:32.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:32.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:32.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:32.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:32.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:32.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:32.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:32.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:32.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:32.858 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:32.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:32.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:33.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704680 2020-03-11T22:51:33.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704680 '!=' null 2020-03-11T22:51:33.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:33.560 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.560 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:33.560 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.652 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:33.652 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:33.652 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:33.652 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:33.653 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:33.654 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:33.655 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:33.656 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:33.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:33.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:33.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:33.761 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:33.761 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:33.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:33.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:33.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:33.762 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:34.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:34.122 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:34.122 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:34.122 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:34.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:34.658 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:34.659 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415447 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415447 '!=' null 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:35.140 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:35.232 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:35.232 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:35.232 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:35.233 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:35.234 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:35.236 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:35.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:35.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:35.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:35.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:35.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:35.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:35.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:35.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:35.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:35.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:35.700 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:35.700 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:35.701 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:36.180 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:36.182 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:36.182 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:36.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415447 2020-03-11T22:51:36.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415447 '!=' null 2020-03-11T22:51:36.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:36.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:36.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.752 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:36.752 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:36.753 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:36.754 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:36.755 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:36.756 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:36.757 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:36.757 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:36.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:36.861 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:36.862 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:36.863 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:37.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:37.241 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:37.241 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:37.241 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:37.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:37.718 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:37.811 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:37.812 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:37.813 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:37.814 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:37.815 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:37.816 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:37.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:37.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:37.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:37.920 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:37.920 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:37.921 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:37.921 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:37.921 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:37.921 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:38.283 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:38.284 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:38.284 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:38.284 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:38.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:38.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:38.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:38.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:38.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:38.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:38.769 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:38.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:38.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27520532 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27520532 '!=' null 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:39.264 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:39.356 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:39.356 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:39.357 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:39.358 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:39.359 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:39.360 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:39.361 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:39.361 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:39.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:39.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:39.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:39.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:39.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:39.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:39.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:39.467 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:39.467 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:39.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:39.825 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:39.825 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:39.826 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:40.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:40.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:40.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:40.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:40.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:40.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:40.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:40.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:40.306 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:40.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13105052 2020-03-11T22:51:40.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13105052 '!=' null 2020-03-11T22:51:40.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:40.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:40.888 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:40.888 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:40.979 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:40.979 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:40.980 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:40.981 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:40.982 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:40.983 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:40.984 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:40.984 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:40.984 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:40.987 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:41.088 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:41.089 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:41.090 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:41.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:41.450 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:41.451 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:41.451 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:41.970 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:41.970 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:41.970 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:41.971 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:42.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13105052 2020-03-11T22:51:42.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13105052 '!=' null 2020-03-11T22:51:42.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:42.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:42.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:42.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:42.590 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:42.590 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:42.590 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:42.590 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:42.590 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:42.591 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:42.593 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:42.594 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:42.594 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:42.594 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:42.594 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:42.594 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:42.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:42.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:42.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:42.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:43.057 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:43.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:43.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:43.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:43.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:43.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:43.537 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:44.130 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23659171 2020-03-11T22:51:44.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23659171 '!=' null 2020-03-11T22:51:44.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:44.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.131 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:44.131 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.223 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:44.223 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:44.223 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:44.223 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:44.223 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:44.224 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:44.225 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:44.226 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:44.227 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:44.227 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:44.227 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:44.227 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:44.227 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:44.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:44.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:44.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:44.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:44.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:44.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:44.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:44.695 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:45.177 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:45.177 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:45.177 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:45.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:45.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20490135 2020-03-11T22:51:45.743 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20490135 '!=' null 2020-03-11T22:51:45.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:45.744 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:45.744 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:45.744 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:45.835 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:45.835 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:45.835 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:45.835 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:45.836 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:45.837 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:45.838 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:45.839 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:45.840 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:45.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:45.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:45.943 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:45.944 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:45.945 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:45.945 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:45.945 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:45.945 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:45.945 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:46.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:46.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:46.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:46.300 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:46.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:46.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:46.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:46.777 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:47.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830950 2020-03-11T22:51:47.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830950 '!=' null 2020-03-11T22:51:47.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:47.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:47.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:47.256 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:47.347 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:47.348 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:47.349 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:47.350 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:47.350 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:47.350 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:47.350 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:47.350 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:47.351 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:47.351 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:47.351 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:47.351 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:47.351 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:47.354 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:47.455 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:47.455 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:47.456 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:47.456 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:47.456 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:47.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:47.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:47.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:47.816 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:47.817 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:47.817 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:47.817 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:48.309 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:48.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:48.311 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:48.311 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:48.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094201 2020-03-11T22:51:48.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094201 '!=' null 2020-03-11T22:51:48.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:48.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:48.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:48.851 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:48.942 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:48.942 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:48.942 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:48.942 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:48.942 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:48.943 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:48.944 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:48.945 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:48.946 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:48.946 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:48.946 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:48.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:49.050 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:49.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:49.051 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:49.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:49.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:49.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:49.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:49.052 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:49.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:49.412 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:49.412 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:49.412 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:49.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:49.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:49.902 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:49.903 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:50.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094201 2020-03-11T22:51:50.448 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094201 '!=' null 2020-03-11T22:51:50.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:50.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:50.449 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:50.449 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:50.541 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:50.541 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:50.542 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:50.543 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:50.544 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:50.545 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:50.548 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:50.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:50.650 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:50.651 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:51.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:51.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:51.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:51.016 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:51.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:51.490 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:51.491 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:51.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20093927 2020-03-11T22:51:51.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20093927 '!=' null 2020-03-11T22:51:51.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:51.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:51.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:51.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:52.084 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:52.084 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:52.085 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:52.086 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:52.088 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:52.091 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:52.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:52.193 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:52.194 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:52.194 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:52.194 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:52.194 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:52.194 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:52.195 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:52.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:52.550 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:52.551 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:52.551 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:53.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:53.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:53.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:53.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:53.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:53.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:53.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:53.035 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:53.035 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704051 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704051 '!=' null 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [ 2020-03-11T22:51:53.607 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: "priority": 110, 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: "items": [ 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:53.608 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "3.0", 2020-03-11T22:51:53.609 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 151, 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:53.610 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:53.611 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:53.611 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:53.611 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:53.611 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:53.614 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:53.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:53.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:53.716 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:53.716 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:53.716 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:53.716 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:53.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:53.717 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:54.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:54.076 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:54.076 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:54.076 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:54.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:54.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:54.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:54.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:54.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:54.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:54.567 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:54.567 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:54.567 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:55.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220360 2020-03-11T22:51:55.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220360 '!=' null 2020-03-11T22:51:55.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:55.099 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:55.099 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:55.099 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:55.191 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:55.192 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:55.194 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:55.194 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:55.194 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:55.194 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:55.194 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:55.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:55.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:55.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:55.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:55.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:55.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:55.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:55.673 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:56.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:56.184 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:56.185 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27520172 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27520172 '!=' null 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:56.770 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:56.862 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:56.862 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:56.863 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:56.864 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:56.865 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:56.865 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:56.865 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:56.865 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:56.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:56.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:56.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:56.972 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:57.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:57.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:57.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:57.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:57.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:57.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:51:57.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:57.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:51:57.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.851 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:57.851 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.904 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:57.904 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:57.904 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:57.904 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:57.904 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:57.905 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:57.906 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:57.910 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:58.017 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:58.019 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:58.019 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:58.020 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:58.020 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:58.021 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:58.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:58.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:58.368 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:58.368 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:58.369 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:58.369 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:58.849 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:58.849 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:58.849 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:58.850 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:59.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27520755 2020-03-11T22:51:59.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27520755 '!=' null 2020-03-11T22:51:59.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:59.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:59.335 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:59.335 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:59.426 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:51:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:51:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:51:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:51:59.429 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:51:59.433 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:59.534 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:59.534 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:59.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:59.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:59.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:59.536 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:59.537 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:59.537 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:59.892 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:59.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:59.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:59.893 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:00.371 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:00.372 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:01.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13105120 2020-03-11T22:52:01.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13105120 '!=' null 2020-03-11T22:52:01.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:01.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:01.165 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:01.166 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:01.257 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:01.257 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:01.257 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:01.257 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:01.257 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:01.258 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:01.259 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:01.260 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:01.260 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:01.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:01.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:01.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:01.366 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:01.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:01.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:01.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:01.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:01.367 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:01.724 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:01.724 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:01.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:01.725 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:02.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:02.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:02.205 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:02.206 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:02.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831315 2020-03-11T22:52:02.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831315 '!=' null 2020-03-11T22:52:02.764 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:02.765 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:02.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:02.765 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:02.857 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:02.857 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:02.858 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:02.859 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:02.860 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:02.860 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:02.860 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:02.860 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:02.860 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:02.864 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:02.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:02.968 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:02.968 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:02.968 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:02.968 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:02.969 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:02.969 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:02.969 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:03.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:03.331 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:03.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:03.332 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:03.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:03.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:03.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:03.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:03.821 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:03.821 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:03.821 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:03.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:03.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:04.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831315 2020-03-11T22:52:04.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831315 '!=' null 2020-03-11T22:52:04.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:04.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.416 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:04.416 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:04.507 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:04.508 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:04.510 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:04.514 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:04.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:04.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:04.616 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:04.616 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:04.616 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:04.617 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:04.617 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:04.617 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:04.976 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:04.976 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:04.977 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:04.977 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:05.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:05.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:05.474 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:05.475 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:05.949 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25523208 2020-03-11T22:52:05.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25523208 '!=' null 2020-03-11T22:52:05.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:05.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:05.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:05.950 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:06.042 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:06.043 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:06.044 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:06.049 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:06.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:06.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:06.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:06.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:06.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:06.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:06.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:06.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:06.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:06.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:06.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:06.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:07.000 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:07.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:07.002 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:07.002 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:07.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31523746 2020-03-11T22:52:07.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31523746 '!=' null 2020-03-11T22:52:07.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:07.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:07.619 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:07.619 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:07.711 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:07.712 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:07.714 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:07.714 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:07.714 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:07.714 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:07.714 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:07.718 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:07.820 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:07.821 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:08.177 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:08.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:08.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:08.178 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:08.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:08.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:08.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:08.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:08.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:08.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:08.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:08.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:08.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:09.141 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220663 2020-03-11T22:52:09.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220663 '!=' null 2020-03-11T22:52:09.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:09.142 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:09.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:09.142 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:09.233 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:09.233 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:09.234 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:09.235 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:09.236 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:09.236 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:09.236 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:09.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:09.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:09.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:09.702 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:09.703 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:09.703 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:09.703 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:10.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220673 2020-03-11T22:52:10.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220673 '!=' null 2020-03-11T22:52:10.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:10.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:10.772 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:10.773 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:10.864 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:10.865 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:10.866 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:10.867 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:10.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:10.972 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:10.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:10.973 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:10.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:10.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:10.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:10.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:10.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:11.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:11.449 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:11.449 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:11.450 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:11.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:11.940 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:11.940 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:11.940 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:12.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220673 2020-03-11T22:52:12.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220673 '!=' null 2020-03-11T22:52:12.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:12.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:12.459 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:12.459 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:12.552 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:12.552 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:12.552 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:12.552 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:12.552 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:12.553 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:12.553 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:12.553 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:12.553 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:12.555 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:12.555 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:12.555 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:12.555 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:12.556 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:12.556 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:12.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:12.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:12.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:13.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:13.022 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:13.023 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:13.500 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:13.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:13.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:13.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:13.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:13.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:13.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:13.502 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:13.502 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:14.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704699 2020-03-11T22:52:14.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704699 '!=' null 2020-03-11T22:52:14.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:14.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:14.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.150 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:14.150 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:14.150 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:14.151 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:14.151 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:14.151 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:14.151 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:14.152 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:14.152 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:14.152 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:14.152 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:14.152 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:14.153 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:14.153 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:14.153 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:14.153 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:14.154 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:14.154 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:14.154 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:14.154 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:14.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:14.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:14.259 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:14.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:14.259 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:14.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:14.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:14.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:14.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:14.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:14.618 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:14.618 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:14.618 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:15.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:15.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:15.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220672 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220672 '!=' null 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:15.787 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:15.788 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:15.789 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:15.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:15.896 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:15.897 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:15.897 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:16.260 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:16.260 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:16.261 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:16.261 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:16.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:16.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:16.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:16.759 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831053 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831053 '!=' null 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:17.299 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.390 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:17.390 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [ 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "item": "1.0", 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "prio": 110, 2020-03-11T22:52:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: "can_preempt": true 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:17.392 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:17.393 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:17.393 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:17.393 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:17.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:17.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:17.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:17.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:17.865 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:17.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:17.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:17.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:18.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:18.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:18.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:18.345 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24113629 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24113629 '!=' null 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:18.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:18.948 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:18.948 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:18.948 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:18.948 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:18.949 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:18.950 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:18.950 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:18.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:19.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:19.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:19.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:19.057 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:19.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:19.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:19.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:19.058 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:19.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:19.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:19.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:19.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:19.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:19.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:19.889 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:19.890 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:20.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24113629 2020-03-11T22:52:20.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24113629 '!=' null 2020-03-11T22:52:20.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:20.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.398 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:20.399 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:20.490 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:20.492 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:20.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:20.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:20.600 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:20.600 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:20.600 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:20.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:20.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:20.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:20.601 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:20.956 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:20.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:20.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:20.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:21.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:21.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:21.437 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:21.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415814 2020-03-11T22:52:21.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415814 '!=' null 2020-03-11T22:52:21.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:21.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:21.920 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:21.920 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:22.011 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:22.011 INFO:tasks.workunit.client.0.smithi045.stdout: "local_reservations": { 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "remote_reservations": { 2020-03-11T22:52:22.012 INFO:tasks.workunit.client.0.smithi045.stdout: "max_allowed": 1, 2020-03-11T22:52:22.013 INFO:tasks.workunit.client.0.smithi045.stdout: "min_priority": 0, 2020-03-11T22:52:22.013 INFO:tasks.workunit.client.0.smithi045.stdout: "queues": [], 2020-03-11T22:52:22.013 INFO:tasks.workunit.client.0.smithi045.stdout: "in_progress": [] 2020-03-11T22:52:22.013 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:22.013 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:22.018 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:22.120 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:22.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:22.122 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:22.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:52:22.475 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:22.476 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:22.476 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:22.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:52:22.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:22.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:22.950 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:52:23.310 INFO:tasks.workunit.client.0.smithi045.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:52:23.310 INFO:tasks.workunit.client.0.smithi045.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:33.316292+0000 93'150 117:342 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:51.074198+0000 0'0 2020-03-11T22:48:51.074198+0000 0 2020-03-11T22:52:23.310 INFO:tasks.workunit.client.0.smithi045.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:18.637311+0000 93'150 118:350 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:37.115158+0000 0'0 2020-03-11T22:48:37.115158+0000 0 2020-03-11T22:52:23.310 INFO:tasks.workunit.client.0.smithi045.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:55.828515+0000 93'150 117:347 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:41.841266+0000 0'0 2020-03-11T22:48:41.841266+0000 0 2020-03-11T22:52:23.311 INFO:tasks.workunit.client.0.smithi045.stdout: 2020-03-11T22:52:23.311 INFO:tasks.workunit.client.0.smithi045.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:52:23.311 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:52:23.320 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:52:23.321 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56115/ceph-osd.1.asok 2020-03-11T22:52:23.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:23.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56115/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:52:23.415 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: "pgs": [ 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: "pg": "7.0", 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: "history": [ 2020-03-11T22:52:23.416 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "62", 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.535753+0000", 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.535765+0000" 2020-03-11T22:52:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.535771+0000", 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.723142+0000" 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.418 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.723144+0000", 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.723213+0000" 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.723213+0000", 2020-03-11T22:52:23.419 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.723227+0000" 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.723227+0000", 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.952768+0000" 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.420 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.535768+0000", 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:51.952789+0000" 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.421 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.953012+0000", 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:52.270362+0000" 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:52.270362+0000", 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:52.270430+0000" 2020-03-11T22:52:23.422 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:52.270430+0000", 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019170+0000" 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.423 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.952789+0000", 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019207+0000" 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.535765+0000", 2020-03-11T22:52:23.424 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019237+0000" 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:51.535750+0000", 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019245+0000" 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "63", 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019246+0000", 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019664+0000" 2020-03-11T22:52:23.426 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "91", 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.427 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019683+0000", 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.019712+0000" 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019730+0000", 2020-03-11T22:52:23.428 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.078656+0000" 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.078657+0000", 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.078764+0000" 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.429 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.078764+0000", 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.078777+0000" 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.078777+0000", 2020-03-11T22:52:23.430 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.307030+0000" 2020-03-11T22:52:23.431 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.431 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.431 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.431 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019721+0000", 2020-03-11T22:52:23.431 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.307043+0000" 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.307169+0000", 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.628106+0000" 2020-03-11T22:52:23.432 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.628107+0000", 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:53.628156+0000" 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.628157+0000", 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.040597+0000" 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.307043+0000", 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.040615+0000" 2020-03-11T22:52:23.434 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019713+0000", 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.040631+0000" 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.435 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:53.019665+0000", 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.040635+0000" 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "92", 2020-03-11T22:52:23.436 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.040635+0000", 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041027+0000" 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.437 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "98", 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041034+0000", 2020-03-11T22:52:23.438 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041057+0000" 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041070+0000", 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.068698+0000" 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.439 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.068699+0000", 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.068764+0000" 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.068764+0000", 2020-03-11T22:52:23.440 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.068776+0000" 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.068776+0000", 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.341774+0000" 2020-03-11T22:52:23.441 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041063+0000", 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.341786+0000" 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.341849+0000", 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.351721+0000" 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.351721+0000", 2020-03-11T22:52:23.443 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.351777+0000" 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.351778+0000", 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.363645+0000" 2020-03-11T22:52:23.444 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.341786+0000", 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.363672+0000" 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.445 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041057+0000", 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.363726+0000" 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041027+0000", 2020-03-11T22:52:23.446 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.363737+0000" 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "99", 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.447 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.363738+0000", 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.364127+0000" 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "100", 2020-03-11T22:52:23.448 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364131+0000", 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.364149+0000" 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.449 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364159+0000", 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.364785+0000" 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364785+0000", 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.364849+0000" 2020-03-11T22:52:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.451 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.451 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.451 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364153+0000", 2020-03-11T22:52:23.451 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.364860+0000" 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364861+0000", 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.648042+0000" 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.453 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364149+0000", 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.648077+0000" 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.454 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.364127+0000", 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.648089+0000" 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "100", 2020-03-11T22:52:23.455 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648090+0000", 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.648538+0000" 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.456 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "113", 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648548+0000", 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.648580+0000" 2020-03-11T22:52:23.457 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648598+0000", 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.673462+0000" 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.673468+0000", 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.673551+0000" 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.673552+0000", 2020-03-11T22:52:23.459 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.673573+0000" 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.673573+0000", 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.847828+0000" 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.460 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648588+0000", 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.847842+0000" 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.848115+0000", 2020-03-11T22:52:23.461 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.924899+0000" 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.924900+0000", 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.925132+0000" 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.462 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.925133+0000", 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:22.925778+0000" 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:23.463 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.925779+0000", 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:54.189282+0000" 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:54.189283+0000", 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:54.189631+0000" 2020-03-11T22:52:23.464 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:54.189631+0000", 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:06.561166+0000" 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.465 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:06.561166+0000", 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:06.561575+0000" 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:06.561576+0000", 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.600402+0000" 2020-03-11T22:52:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:32.600403+0000", 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.600656+0000" 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.467 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:32.600656+0000", 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.713777+0000" 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.847843+0000", 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.713805+0000" 2020-03-11T22:52:23.468 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648581+0000", 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.713822+0000" 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.469 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:22.648538+0000", 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.713828+0000" 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.470 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "113", 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:32.713829+0000", 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.714079+0000" 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.471 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: "pg": "3.0", 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: "history": [ 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.472 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "38", 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.652877+0000", 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.652887+0000" 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.473 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.652893+0000", 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.701422+0000" 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.474 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.701422+0000", 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.701507+0000" 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.701508+0000", 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.701522+0000" 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.701522+0000", 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.801524+0000" 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.652890+0000", 2020-03-11T22:52:23.476 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:42.801546+0000" 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.801822+0000", 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.346720+0000" 2020-03-11T22:52:23.477 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.801546+0000", 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.346737+0000" 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.478 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.652887+0000", 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.346754+0000" 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:42.652874+0000", 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.346758+0000" 2020-03-11T22:52:23.479 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "40", 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.480 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.346759+0000", 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.347007+0000" 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.481 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "89", 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.347011+0000", 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.347024+0000" 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.482 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.347030+0000", 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.444146+0000" 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.444147+0000", 2020-03-11T22:52:23.483 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.444279+0000" 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.444279+0000", 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.444304+0000" 2020-03-11T22:52:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.444305+0000", 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.592632+0000" 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.485 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.347027+0000", 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.592639+0000" 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.592752+0000", 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.919109+0000" 2020-03-11T22:52:23.486 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.919109+0000", 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:44.919162+0000" 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.487 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.919163+0000", 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041400+0000" 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.592640+0000", 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041429+0000" 2020-03-11T22:52:23.488 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.347024+0000", 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041451+0000" 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.489 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:44.347007+0000", 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041460+0000" 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.490 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "92", 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041460+0000", 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041756+0000" 2020-03-11T22:52:23.491 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "105", 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041766+0000", 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.041778+0000" 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041789+0000", 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.184330+0000" 2020-03-11T22:52:23.493 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.184330+0000", 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.184390+0000" 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.494 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.184390+0000", 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.184404+0000" 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.184404+0000", 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.342002+0000" 2020-03-11T22:52:23.495 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041782+0000", 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.342010+0000" 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.496 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.342099+0000", 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.360034+0000" 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.360034+0000", 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:49:18.360073+0000" 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.360074+0000", 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.445626+0000" 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.498 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.342012+0000", 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.445654+0000" 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041778+0000", 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.445696+0000" 2020-03-11T22:52:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:49:18.041757+0000", 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.445702+0000" 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.500 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "106", 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.445702+0000", 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.446025+0000" 2020-03-11T22:52:23.501 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "107", 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.502 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446031+0000", 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.446045+0000" 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446053+0000", 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.446506+0000" 2020-03-11T22:52:23.503 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446507+0000", 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.446563+0000" 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.504 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446049+0000", 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.446571+0000" 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446572+0000", 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.708456+0000" 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446046+0000", 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.708489+0000" 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.446026+0000", 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.708494+0000" 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "107", 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708494+0000", 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.708887+0000" 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "115", 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.509 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708897+0000", 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.708925+0000" 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.510 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708942+0000", 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.786647+0000" 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.786648+0000", 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.786721+0000" 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.511 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.786722+0000", 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.786739+0000" 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.512 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.786739+0000", 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.875956+0000" 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708933+0000", 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:41.875966+0000" 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.513 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.876173+0000", 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:42.569440+0000" 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:23.514 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:42.569441+0000", 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:54.189446+0000" 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:54.189447+0000", 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:54.189915+0000" 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.515 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:54.189916+0000", 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:06.561017+0000" 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:06.561018+0000", 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:06.561262+0000" 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:06.561262+0000", 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.600928+0000" 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:32.600928+0000", 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:32.601400+0000" 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:23.518 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:32.601400+0000", 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.025648+0000" 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:55.025649+0000", 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.025901+0000" 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.519 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:55.025901+0000", 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.511293+0000" 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.875967+0000", 2020-03-11T22:52:23.520 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.511318+0000" 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708926+0000", 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.511346+0000" 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.521 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:41.708887+0000", 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.511359+0000" 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.522 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "115", 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:55.511360+0000", 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.511678+0000" 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.523 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: "pg": "1.0", 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: "history": [ 2020-03-11T22:52:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "26", 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Initial", 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851113+0000", 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.851220+0000" 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.525 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "26", 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.526 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851221+0000", 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.851251+0000" 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "28", 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851260+0000", 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.851270+0000" 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.528 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851277+0000", 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.916958+0000" 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.916959+0000", 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.917022+0000" 2020-03-11T22:52:23.529 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.917022+0000", 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:37.917034+0000" 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.530 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.917034+0000", 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:38.068038+0000" 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851274+0000", 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:38.068047+0000" 2020-03-11T22:52:23.531 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:38.068252+0000", 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:38.683698+0000" 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.532 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:38.683699+0000", 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:38.683758+0000" 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:38.683758+0000", 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.092845+0000" 2020-03-11T22:52:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:38.068047+0000", 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.092864+0000" 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.534 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851271+0000", 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.092879+0000" 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:37.851251+0000", 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.092883+0000" 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.535 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "29", 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.536 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.092883+0000", 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.093100+0000" 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "102", 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.537 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.093104+0000", 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.093118+0000" 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.538 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.093124+0000", 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.118717+0000" 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.118717+0000", 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.118783+0000" 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.539 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.118783+0000", 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.118796+0000" 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.118796+0000", 2020-03-11T22:52:23.540 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.293824+0000" 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.093121+0000", 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.293834+0000" 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.293976+0000", 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.319373+0000" 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.319373+0000", 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:48:39.319477+0000" 2020-03-11T22:52:23.542 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.319477+0000", 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355005+0000" 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.543 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.293835+0000", 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355031+0000" 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.544 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.093118+0000", 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355071+0000" 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:48:39.093101+0000", 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355076+0000" 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.545 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.546 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.546 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.546 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "102", 2020-03-11T22:52:23.546 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.546 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.547 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.547 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355076+0000", 2020-03-11T22:52:23.547 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355355+0000" 2020-03-11T22:52:23.547 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.547 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "103", 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.548 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355361+0000", 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.355374+0000" 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355383+0000", 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.397758+0000" 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.397758+0000", 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.397818+0000" 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.550 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355378+0000", 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.397826+0000" 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.397826+0000", 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.656955+0000" 2020-03-11T22:52:23.551 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355374+0000", 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.656974+0000" 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.355356+0000", 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.656978+0000" 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.553 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "103", 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.656979+0000", 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.657215+0000" 2020-03-11T22:52:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "117", 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.555 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Start", 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.657224+0000", 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.657255+0000" 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.657273+0000", 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.712030+0000" 2020-03-11T22:52:23.556 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.712031+0000", 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.712116+0000" 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.712117+0000", 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.712141+0000" 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.712141+0000", 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.874298+0000" 2020-03-11T22:52:23.558 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.657264+0000", 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:31.874313+0000" 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.874580+0000", 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:50:32.607948+0000" 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:23.560 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:32.607949+0000", 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.026164+0000" 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:55.026165+0000", 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:51:55.026628+0000" 2020-03-11T22:52:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:51:55.026629+0000", 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.071081+0000" 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.562 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:52:18.071081+0000", 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.071265+0000" 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:52:18.071265+0000", 2020-03-11T22:52:23.563 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.323873+0000" 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.874313+0000", 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.323896+0000" 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started/Primary", 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.657255+0000", 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.323915+0000" 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Started", 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:50:31.657215+0000", 2020-03-11T22:52:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.323921+0000" 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: "epoch": "117", 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: "states": [ 2020-03-11T22:52:23.566 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: "state": "Reset", 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: "enter": "2020-03-11T22:52:18.323922+0000", 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: "exit": "2020-03-11T22:52:18.324201+0000" 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:52:23.568 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:52:23.568 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:52:23.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:52:23.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:52:23.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:52:23.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:52:23.569 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:52:23.570 INFO:tasks.workunit.client.0.smithi045.stdout:TEST PASSED 2020-03-11T22:52:23.933 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test1' does not exist 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:52:24.545 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test3' does not exist 2020-03-11T22:52:24.553 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:52:24.553 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:52:24.553 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:52:25.035 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test7' does not exist 2020-03-11T22:52:25.043 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:52:25.043 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:25.044 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:25.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:25.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:25.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:25.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:25.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:25.154 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:25.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:25.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:25.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:25.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:25.160 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:25.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:25.161 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:25.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:25.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:25.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:25.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:25.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:25.165 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:25.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:25.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:25.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:25.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.192 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:52:25.192 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56115 2020-03-11T22:52:25.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:25.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:25.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:52:25.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:52:25.194 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:25.195 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:25.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:25.196 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:25.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:25.197 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:25.197 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:25.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:25.198 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:25.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:25.201 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:25.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:52:25.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:25.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:25.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56115 2020-03-11T22:52:25.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56115 2020-03-11T22:52:25.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:25.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:25.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:52:25.205 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:52:25.205 INFO:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:52:25.273 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:52:25.273 INFO:teuthology.orchestra.run.smithi045:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:52:25.305 INFO:tasks.workunit.client.0.smithi045.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:52:25.313 INFO:tasks.workunit.client.0.smithi045.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:52:25.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:25.313 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:52:25.314 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:52:25.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none ' 2020-03-11T22:52:25.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:25.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:52:25.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:52:25.318 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:52:25.318 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_backfill_log_2 2020-03-11T22:52:25.318 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_recovery_1 2020-03-11T22:52:25.318 INFO:tasks.workunit.client.0.smithi045.stderr:TEST_recovery_2' 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:52:25.319 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:25.320 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:25.321 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:25.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:25.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:25.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:25.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76792 2020-03-11T22:52:25.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:25.325 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:25.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:52:25.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:52:25.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.326 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:25.326 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76792 2020-03-11T22:52:25.328 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:52:25.330 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:52:25.331 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:52:25.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:25.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:25.595 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:25.595 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.595 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:25.595 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:25.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:52:25.754 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:52:25.755 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:25.756 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76792/ceph-mon.a.asok 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:25.757 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76792/ceph-mon.a.asok config get fsid 2020-03-11T22:52:25.847 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:52:25.847 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:25.848 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:25.849 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:25.849 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:25.849 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76792/ceph-mon.a.asok 2020-03-11T22:52:25.849 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:25.849 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76792/ceph-mon.a.asok config get mon_host 2020-03-11T22:52:25.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:52:25.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:25.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:52:25.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:52:25.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:52:25.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:52:25.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:26.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:52:26.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:26.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:26.898 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:52:26.899 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:52:26.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:26.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:52:26.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:26.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:26.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:26.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:26.907 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:26.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:26.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:52:26.908 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:26.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ec0b9172-49d5-414b-beea-594ec23b59d3 2020-03-11T22:52:26.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 ec0b9172-49d5-414b-beea-594ec23b59d3' 2020-03-11T22:52:26.909 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:26.909 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 ec0b9172-49d5-414b-beea-594ec23b59d3 2020-03-11T22:52:26.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCqa2leY0bsNhAAcROgmNCBG+5+5I/ITTTQhg== 2020-03-11T22:52:26.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCqa2leY0bsNhAAcROgmNCBG+5+5I/ITTTQhg=="}' 2020-03-11T22:52:26.924 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ec0b9172-49d5-414b-beea-594ec23b59d3 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:27.271 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:27.280 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:27.281 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCqa2leY0bsNhAAcROgmNCBG+5+5I/ITTTQhg== --osd-uuid ec0b9172-49d5-414b-beea-594ec23b59d3 2020-03-11T22:52:27.306 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.302+0000 7fed64d14ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:27.308 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.304+0000 7fed64d14ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.305+0000 7fed64d14ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.306+0000 7fed64d14ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.306+0000 7fed64d14ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.306+0000 7fed64d14ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:27.353 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:27.350+0000 7fed64d14ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:52:30.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:52:30.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:30.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:52:30.566 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:30.566 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:52:31.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:52:31.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:31.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:31.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:31.046 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:31.046 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:52:31.068 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:31.064+0000 7fb5bb620ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:31.070 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:31.066+0000 7fb5bb620ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:31.071 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:31.068+0000 7fb5bb620ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:31.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:31.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:31.426 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:31.426 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:31.588 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:31.584+0000 7fb5bb620ec0 -1 Falling back to public interface 2020-03-11T22:52:31.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:32.391 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:32.386+0000 7fb5bb620ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:52:32.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:32.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:32.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:32.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:32.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:32.790 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:33.228 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77224,v1:127.0.0.1:6803/77224] [v2:127.0.0.1:6804/77224,v1:127.0.0.1:6805/77224] exists,up ec0b9172-49d5-414b-beea-594ec23b59d3 2020-03-11T22:52:33.228 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 cd564565-b2e8-413b-a4e5-a6c28b85726e 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:52:33.229 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:33.230 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:33.231 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:33.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=cd564565-b2e8-413b-a4e5-a6c28b85726e 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 cd564565-b2e8-413b-a4e5-a6c28b85726e' 2020-03-11T22:52:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:33.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCxa2lekn5DDhAAzIYMRXaCe4t5yet5HaHlXg== 2020-03-11T22:52:33.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCxa2lekn5DDhAAzIYMRXaCe4t5yet5HaHlXg=="}' 2020-03-11T22:52:33.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new cd564565-b2e8-413b-a4e5-a6c28b85726e -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:33.858 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:33.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:33.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCxa2lekn5DDhAAzIYMRXaCe4t5yet5HaHlXg== --osd-uuid cd564565-b2e8-413b-a4e5-a6c28b85726e 2020-03-11T22:52:33.893 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.890+0000 7f95421bfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:33.927 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.924+0000 7f95421bfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:33.929 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.925+0000 7f95421bfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:33.930 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.926+0000 7f95421bfec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:33.930 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.926+0000 7f95421bfec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:33.930 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.926+0000 7f95421bfec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:33.994 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:33.991+0000 7f95421bfec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:52:37.213 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:52:37.213 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:37.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:52:37.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:37.214 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:52:37.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:52:37.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:37.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:37.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:37.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:37.702 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:52:37.723 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:37.720+0000 7f8bb8adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:37.725 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:37.722+0000 7f8bb8adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:37.727 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:37.723+0000 7f8bb8adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:38.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:38.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:38.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:38.098 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:38.250 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:38.246+0000 7f8bb8adcec0 -1 Falling back to public interface 2020-03-11T22:52:38.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:39.008 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:39.004+0000 7f8bb8adcec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:52:39.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:39.474 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:39.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:39.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:39.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:39.475 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:39.872 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77579,v1:127.0.0.1:6811/77579] [v2:127.0.0.1:6812/77579,v1:127.0.0.1:6813/77579] exists,up cd564565-b2e8-413b-a4e5-a6c28b85726e 2020-03-11T22:52:39.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:39.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:39.874 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:39.875 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:39.876 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:39.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:39.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:52:39.878 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:39.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4d5d0206-0b9c-4699-87c3-916e66d116a9 2020-03-11T22:52:39.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 4d5d0206-0b9c-4699-87c3-916e66d116a9' 2020-03-11T22:52:39.879 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:39.879 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 4d5d0206-0b9c-4699-87c3-916e66d116a9 2020-03-11T22:52:39.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC3a2le3BvMNBAAtpqWQxSmJRPUhn0w8ywqOg== 2020-03-11T22:52:39.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC3a2le3BvMNBAAtpqWQxSmJRPUhn0w8ywqOg=="}' 2020-03-11T22:52:39.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4d5d0206-0b9c-4699-87c3-916e66d116a9 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:52:40.459 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:40.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3a2le3BvMNBAAtpqWQxSmJRPUhn0w8ywqOg== --osd-uuid 4d5d0206-0b9c-4699-87c3-916e66d116a9 2020-03-11T22:52:40.485 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.481+0000 7f17fb61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.486 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.483+0000 7f17fb61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.487 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.484+0000 7f17fb61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.488 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.484+0000 7f17fb61cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:40.488 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.484+0000 7f17fb61cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:40.488 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.484+0000 7f17fb61cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:40.594 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:40.591+0000 7f17fb61cec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:52:43.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:52:43.787 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:43.788 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:52:43.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:52:43.788 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:44.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:52:44.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:44.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:44.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:44.279 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:44.279 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:52:44.301 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:44.297+0000 7f57389a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.302 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:44.299+0000 7f57389a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.304 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:44.300+0000 7f57389a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:52:44.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:44.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:44.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:44.660 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:44.819 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:44.815+0000 7f57389a7ec0 -1 Falling back to public interface 2020-03-11T22:52:45.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:45.577 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:45.574+0000 7f57389a7ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:52:46.025 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:46.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:46.026 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:46.026 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:46.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:46.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:46.505 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77934,v1:127.0.0.1:6819/77934] [v2:127.0.0.1:6820/77934,v1:127.0.0.1:6821/77934] exists,up 4d5d0206-0b9c-4699-87c3-916e66d116a9 2020-03-11T22:52:46.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:46.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:46.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:46.508 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:46.509 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:46.510 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:46.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:46.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:46.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:46.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:52:46.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:46.512 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7f84834a-3240-4508-aaa1-7564126ed5a1 2020-03-11T22:52:46.512 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 7f84834a-3240-4508-aaa1-7564126ed5a1' 2020-03-11T22:52:46.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:46.512 INFO:tasks.workunit.client.0.smithi045.stdout:add osd3 7f84834a-3240-4508-aaa1-7564126ed5a1 2020-03-11T22:52:46.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC+a2leb0/uHhAAq0JR4ay1C365ucj6h0Musg== 2020-03-11T22:52:46.522 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC+a2leb0/uHhAAq0JR4ay1C365ucj6h0Musg=="}' 2020-03-11T22:52:46.522 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7f84834a-3240-4508-aaa1-7564126ed5a1 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:47.067 INFO:tasks.workunit.client.0.smithi045.stdout:3 2020-03-11T22:52:47.077 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:47.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC+a2leb0/uHhAAq0JR4ay1C365ucj6h0Musg== --osd-uuid 7f84834a-3240-4508-aaa1-7564126ed5a1 2020-03-11T22:52:47.102 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.100+0000 7fad13ea3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:47.144 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.141+0000 7fad13ea3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:47.146 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.143+0000 7fad13ea3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:47.147 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.144+0000 7fad13ea3ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:47.147 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.144+0000 7fad13ea3ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:47.147 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.144+0000 7fad13ea3ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:47.211 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:47.208+0000 7fad13ea3ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:52:50.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:52:50.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:50.367 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd3 key to auth repository 2020-03-11T22:52:50.367 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:52:50.367 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:50.854 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.3 2020-03-11T22:52:50.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:52:50.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:50.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:50.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:50.855 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:50.877 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:50.874+0000 7f3487b9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:50.879 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:50.876+0000 7f3487b9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:50.880 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:50.877+0000 7f3487b9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:51.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:52:51.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:51.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:51.222 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:51.397 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:51.395+0000 7f3487b9eec0 -1 Falling back to public interface 2020-03-11T22:52:51.583 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:52.147 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:52.144+0000 7f3487b9eec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:52:52.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:52.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:52.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:52.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:52.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:52.585 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:52.989 INFO:tasks.workunit.client.0.smithi045.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78297,v1:127.0.0.1:6827/78297] [v2:127.0.0.1:6828/78297,v1:127.0.0.1:6829/78297] exists,up 7f84834a-3240-4508-aaa1-7564126ed5a1 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:52:52.990 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:52.991 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:52.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:52.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:52.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:52.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:52.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:52:52.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:52.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=53338f4a-b91a-4d85-9753-a4e8a0c842a9 2020-03-11T22:52:52.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 53338f4a-b91a-4d85-9753-a4e8a0c842a9' 2020-03-11T22:52:52.996 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:52.996 INFO:tasks.workunit.client.0.smithi045.stdout:add osd4 53338f4a-b91a-4d85-9753-a4e8a0c842a9 2020-03-11T22:52:53.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDFa2le8wQwABAAfSG7enoa0R1WIv0svtmbew== 2020-03-11T22:52:53.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDFa2le8wQwABAAfSG7enoa0R1WIv0svtmbew=="}' 2020-03-11T22:52:53.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 53338f4a-b91a-4d85-9753-a4e8a0c842a9 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:53.627 INFO:tasks.workunit.client.0.smithi045.stdout:4 2020-03-11T22:52:53.637 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:53.638 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDFa2le8wQwABAAfSG7enoa0R1WIv0svtmbew== --osd-uuid 53338f4a-b91a-4d85-9753-a4e8a0c842a9 2020-03-11T22:52:53.662 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.659+0000 7f4848b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.664 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.661+0000 7f4848b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.666 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.663+0000 7f4848b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.667 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.664+0000 7f4848b8cec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:53.667 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.664+0000 7f4848b8cec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:53.667 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.664+0000 7f4848b8cec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:53.754 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:53.751+0000 7f4848b8cec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:52:57.022 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:52:57.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd4 key to auth repository 2020-03-11T22:52:57.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:52:57.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:57.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:57.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:57.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:57.517 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.4 2020-03-11T22:52:57.539 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:57.536+0000 7facad128ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.541 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:57.538+0000 7facad128ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.541 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:57.539+0000 7facad128ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:52:57.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:57.878 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:52:57.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:57.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:57.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:57.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:57.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:57.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:57.880 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:52:58.057 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:58.054+0000 7facad128ec0 -1 Falling back to public interface 2020-03-11T22:52:58.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:58.839 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:52:58.837+0000 7facad128ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:52:59.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:59.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:59.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:59.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:59.268 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:59.268 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:52:59.655 INFO:tasks.workunit.client.0.smithi045.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78652,v1:127.0.0.1:6835/78652] [v2:127.0.0.1:6836/78652,v1:127.0.0.1:6837/78652] exists,up 53338f4a-b91a-4d85-9753-a4e8a0c842a9 2020-03-11T22:52:59.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:59.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:59.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:59.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:59.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:52:59.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:59.658 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:59.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:59.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:52:59.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:59.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0df61b1d-158e-477d-9809-ab5f8b7c504a 2020-03-11T22:52:59.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 0df61b1d-158e-477d-9809-ab5f8b7c504a' 2020-03-11T22:52:59.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:59.662 INFO:tasks.workunit.client.0.smithi045.stdout:add osd5 0df61b1d-158e-477d-9809-ab5f8b7c504a 2020-03-11T22:52:59.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDLa2lehQrOJxAAjQk0BEXGnXPPJjkiok3bcw== 2020-03-11T22:52:59.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDLa2lehQrOJxAAjQk0BEXGnXPPJjkiok3bcw=="}' 2020-03-11T22:52:59.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0df61b1d-158e-477d-9809-ab5f8b7c504a -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:00.284 INFO:tasks.workunit.client.0.smithi045.stdout:5 2020-03-11T22:53:00.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:00.295 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDLa2lehQrOJxAAjQk0BEXGnXPPJjkiok3bcw== --osd-uuid 0df61b1d-158e-477d-9809-ab5f8b7c504a 2020-03-11T22:53:00.320 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.317+0000 7f5e73ad3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:00.321 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.319+0000 7f5e73ad3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:00.323 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.320+0000 7f5e73ad3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:00.323 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.321+0000 7f5e73ad3ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:00.324 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.321+0000 7f5e73ad3ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:00.324 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.321+0000 7f5e73ad3ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:00.364 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:00.361+0000 7f5e73ad3ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:53:03.783 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:53:03.784 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:03.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:53:03.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:03.785 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd5 key to auth repository 2020-03-11T22:53:04.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:53:04.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:04.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:04.270 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:04.271 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:04.271 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.5 2020-03-11T22:53:04.293 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:04.290+0000 7fdba881aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:04.351 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:04.348+0000 7fdba881aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:04.352 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:04.349+0000 7fdba881aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:04.679 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:53:04.679 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:04.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:53:04.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:04.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:04.680 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:04.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:04.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:04.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:04.681 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:53:04.868 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:04.865+0000 7fdba881aec0 -1 Falling back to public interface 2020-03-11T22:53:05.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:05.659 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:53:05.656+0000 7fdba881aec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:53:06.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:06.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:06.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:06.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:06.042 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:06.043 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:53:06.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:07.419 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:07.419 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:07.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:53:07.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:07.420 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:07.420 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:53:07.792 INFO:tasks.workunit.client.0.smithi045.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/79010,v1:127.0.0.1:6843/79010] [v2:127.0.0.1:6844/79010,v1:127.0.0.1:6845/79010] exists,up 0df61b1d-158e-477d-9809-ab5f8b7c504a 2020-03-11T22:53:07.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:07.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:07.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:07.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:53:07.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:53:08.300 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:53:08.308 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:53:09.310 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:53:09.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:09.311 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:53:09.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:09.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:53:09.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:09.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:53:10.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.041 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:53:10.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.089 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:53:10.130 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.131 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:53:10.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:53:10.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.264 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:53:10.305 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:53:10.416 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:53:10.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.584 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:53:10.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:53:10.672 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.673 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:53:10.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:53:10.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.789 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:53:10.830 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:10.831 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:53:11.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:53:11.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.144 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:53:11.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.231 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:53:11.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.306 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:53:11.347 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.348 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:53:11.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.389 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:53:11.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:53:11.586 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:53:11.728 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.729 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:53:11.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:53:11.814 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:53:11.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:53:11.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:53:11.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:11.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:53:12.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:53:12.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:53:12.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.322 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:53:12.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.364 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:53:12.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:53:12.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:53:12.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:53:12.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:53:12.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:12.923 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:53:13.006 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:53:13.047 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.047 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:53:13.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:53:13.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.165 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:53:13.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:53:13.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:53:13.531 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.532 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:53:13.573 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.574 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:53:13.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.639 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:53:13.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:53:13.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:53:13.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.877 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:53:13.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.975 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:53:14.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:53:14.057 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.057 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:53:14.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.132 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:53:14.173 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:53:14.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:53:14.365 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.366 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:53:14.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:53:14.610 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:53:14.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:53:14.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.699 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:53:14.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:53:14.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:53:14.940 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:53:15.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:53:15.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.098 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:53:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:53:15.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:53:15.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.332 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:53:15.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.437 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:53:15.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:53:15.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:53:15.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:53:15.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:53:15.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:53:15.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:53:15.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.989 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:53:16.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:53:16.197 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:53:16.241 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:53:16.357 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:53:16.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:53:16.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.450 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:53:16.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:53:16.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:53:16.799 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.800 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:53:16.841 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.842 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:53:16.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:53:16.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.935 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:53:17.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:53:17.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:53:17.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.249 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:53:17.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:53:17.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:53:17.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.426 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:53:17.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.500 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:53:17.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.568 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:53:17.608 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:53:17.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.761 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:53:17.852 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.853 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:53:17.900 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.901 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:53:17.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:53:18.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:53:18.075 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:53:18.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.278 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:53:18.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.370 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:53:18.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:53:18.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:53:18.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.601 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:53:18.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:53:18.770 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.771 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:53:18.887 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.888 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:53:19.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.030 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:53:19.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.076 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:53:19.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.118 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:53:19.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:53:19.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:53:19.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.284 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:53:19.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:53:19.531 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.531 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:53:19.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:53:19.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:53:19.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:53:19.734 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.734 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:53:19.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.785 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:53:19.889 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:53:20.015 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.016 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:53:20.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:53:20.101 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:53:20.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.143 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:53:20.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:53:20.251 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:53:20.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:53:20.502 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:53:20.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.619 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:53:20.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.660 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:53:20.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.735 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:53:20.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:53:20.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:53:20.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:53:21.059 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:53:21.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.102 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:53:21.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.212 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:53:21.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.269 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:53:21.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:53:21.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:53:21.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:53:21.570 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:21.570 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:53:21.910 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:53:21.919 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:53:22.445 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:53:22.453 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:53:23.455 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:53:23.455 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:53:23.456 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:53:23.456 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:53:23.456 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:53:23.457 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:53:23.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:53:23.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:53:23.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:53:23.564 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:23.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:23.973 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:23.973 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:53:23.973 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:53:23.973 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:53:23.973 INFO:tasks.workunit.client.0.smithi045.stderr:4 2020-03-11T22:53:23.974 INFO:tasks.workunit.client.0.smithi045.stderr:5' 2020-03-11T22:53:23.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:23.974 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:23.974 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:24.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:53:24.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:53:24.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:24.208 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:24.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:53:24.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2020-03-11T22:53:24.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:24.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:24.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:53:24.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450' 2020-03-11T22:53:24.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:24.492 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:24.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345929 2020-03-11T22:53:24.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929' 2020-03-11T22:53:24.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:24.623 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:24.752 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:53:24.752 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182408' 2020-03-11T22:53:24.753 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:24.753 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:53:24.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018886 2020-03-11T22:53:24.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182408 5-128849018886' 2020-03-11T22:53:24.882 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:24.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:53:24.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:24.884 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:24.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:53:24.885 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:24.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:53:24.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:53:24.886 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:53:24.887 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:25.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836493 2020-03-11T22:53:25.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:26.267 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:26.268 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:26.652 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836493 2020-03-11T22:53:26.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:27.653 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:53:27.655 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:28.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836493 2020-03-11T22:53:28.035 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:29.036 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 298 -eq 0 ']' 2020-03-11T22:53:29.037 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:29.423 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836493 2020-03-11T22:53:29.423 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:30.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 297 -eq 0 ']' 2020-03-11T22:53:30.425 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:30.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836493 2020-03-11T22:53:30.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:30.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:30.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:30.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:30.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:30.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:30.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:53:30.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:53:30.805 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:53:30.806 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:31.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672971 2020-03-11T22:53:31.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:31.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:53:31.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:31.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:31.225 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:53:31.225 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:31.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:53:31.226 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:53:31.226 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:53:31.227 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:31.593 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509450 2020-03-11T22:53:31.593 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:31.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345929 2020-03-11T22:53:31.594 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:31.595 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:31.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345929 2020-03-11T22:53:31.596 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:31.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345929 2020-03-11T22:53:31.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2020-03-11T22:53:31.597 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 85899345929 2020-03-11T22:53:31.598 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:31.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345929 2020-03-11T22:53:31.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:31.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:53:31.964 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:31.965 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:31.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:53:31.966 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:31.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:53:31.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:53:31.967 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:32.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182410 -lt 107374182408 2020-03-11T22:53:32.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:32.333 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-128849018886 2020-03-11T22:53:32.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:32.334 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:53:32.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-128849018886 2020-03-11T22:53:32.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:32.337 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=128849018886 2020-03-11T22:53:32.337 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2020-03-11T22:53:32.337 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.5 seq 128849018886 2020-03-11T22:53:32.337 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:53:32.730 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 128849018888 -lt 128849018886 2020-03-11T22:53:32.730 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:53:32.731 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:32.731 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:53:33.232 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:33.233 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:33.587 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:33.588 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:33.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:33.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:34.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:34.060 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:53:34.061 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:53:34.061 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:53:34.061 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:34.061 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:34.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:34.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:34.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:34.162 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:34.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:34.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:34.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:34.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:34.516 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:34.517 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:34.517 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:34.517 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:35.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:35.001 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:35.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:35.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:35.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:35.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:35.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:35.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:35.003 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:35.475 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31244 2020-03-11T22:53:35.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31244 '!=' null 2020-03-11T22:53:35.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:35.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:35.476 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:35.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:35.576 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:35.577 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:35.577 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:35.577 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:35.577 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:35.578 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:35.578 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:35.933 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:35.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:35.934 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:35.935 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:36.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:36.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:36.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:36.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:36.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26448 2020-03-11T22:53:36.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26448 '!=' null 2020-03-11T22:53:36.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:36.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:36.881 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:36.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:36.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:36.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:36.982 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:36.983 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:36.983 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:36.983 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:36.983 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:37.333 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:37.334 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:37.335 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:37.335 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:37.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:37.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:37.809 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:37.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:37.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:37.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:37.810 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:37.810 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:37.811 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:38.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26448 2020-03-11T22:53:38.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26448 '!=' null 2020-03-11T22:53:38.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:38.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:38.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:38.424 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:38.425 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:38.426 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:38.777 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:38.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:38.778 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:38.779 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:39.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:39.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:39.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:39.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:39.252 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:39.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:39.253 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:39.253 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:39.253 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:39.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26448 2020-03-11T22:53:39.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26448 '!=' null 2020-03-11T22:53:39.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:39.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:39.726 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:39.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:39.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:39.827 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:39.828 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:39.828 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:39.828 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:39.828 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:39.828 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:40.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:40.179 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:40.180 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:40.180 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:40.655 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:40.656 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:40.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:40.657 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:41.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41397 2020-03-11T22:53:41.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41397 '!=' null 2020-03-11T22:53:41.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:41.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:41.133 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:41.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:41.234 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:41.235 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:41.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:41.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:41.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:41.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:42.077 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:42.078 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:42.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:42.079 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:42.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:42.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:42.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:42.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:42.559 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:42.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:42.659 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:42.660 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:42.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:42.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:42.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:42.661 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:42.662 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:43.014 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:43.015 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:43.015 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:43.015 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:43.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:43.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:43.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:43.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:43.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:43.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:43.489 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:43.489 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:43.490 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:43.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:43.980 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:43.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:43.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:43.981 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:44.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:44.081 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:44.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:44.082 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:44.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:44.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:44.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:44.083 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:44.438 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:44.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:44.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:44.439 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:44.913 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:44.913 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:44.913 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:44.913 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:44.913 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:44.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:44.914 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:44.914 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:44.914 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:45.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45999 2020-03-11T22:53:45.394 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45999 '!=' null 2020-03-11T22:53:45.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:45.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:45.395 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:45.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:45.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:45.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:45.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:45.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:45.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:45.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:45.497 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:45.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:45.858 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:45.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:45.859 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:46.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:46.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:46.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:46.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:46.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:46.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:46.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:46.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:46.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:46.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:46.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:46.926 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:46.927 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:46.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:46.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:46.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:46.928 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:46.929 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:47.282 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:47.282 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:47.283 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:47.283 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:47.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:47.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:47.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:47.759 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:47.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:47.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:47.760 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:47.760 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:47.761 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:48.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:48.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:48.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:48.237 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:48.238 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:48.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:48.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:48.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:48.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:48.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:48.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:48.339 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:48.340 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:48.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:48.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:48.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:48.693 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:49.166 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:49.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:49.168 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:49.168 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:49.700 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:49.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:49.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:49.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:49.701 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:49.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:49.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:49.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:49.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:49.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:49.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:49.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:49.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:50.155 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:50.156 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:50.156 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:50.157 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:50.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:50.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:50.657 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:50.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:50.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:50.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:50.658 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:50.659 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:50.659 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:51.174 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45424 2020-03-11T22:53:51.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45424 '!=' null 2020-03-11T22:53:51.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:51.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:51.175 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:51.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:51.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:51.277 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:51.630 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:51.631 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:51.631 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:51.632 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:52.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:52.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:52.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:52.108 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:52.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:52.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:52.109 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:52.109 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:52.110 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:52.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:52.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:52.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:52.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:52.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:52.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:52.762 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:52.763 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:52.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:52.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:52.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:52.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:52.764 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:53.117 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:53.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:53.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:53.121 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:53.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:53.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:53.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:53.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:53.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:53.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:53.591 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:53.591 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:53.592 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:54.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:53:54.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:53:54.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:54.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:54.067 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:54.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:54.167 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:54.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:54.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:54.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:54.169 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:54.170 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:54.170 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:54.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:54.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:54.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:54.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:54.993 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:54.994 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:54.995 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:54.995 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:55.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22424 2020-03-11T22:53:55.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22424 '!=' null 2020-03-11T22:53:55.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:55.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:55.485 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:55.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:55.585 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:55.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:55.586 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:55.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:55.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:55.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:55.587 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:55.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:55.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:55.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:55.938 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:56.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:56.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:56.417 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:56.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:56.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:56.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:56.418 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:56.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:56.419 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:56.903 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=43986 2020-03-11T22:53:56.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 43986 '!=' null 2020-03-11T22:53:56.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:56.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:56.904 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:57.004 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:57.005 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:57.006 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:57.006 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:57.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:57.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:57.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:57.007 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:57.356 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:57.357 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:57.357 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:57.357 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:57.825 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:53:57.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:57.826 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:57.826 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:57.826 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:53:58.171 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:53:58.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:53:58.179 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:53:58.180 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:58.181 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:58.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:58.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:58.181 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:58.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:58.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:53:58.292 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:58.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:59.424 INFO:tasks.workunit.client.0.smithi045.stdout:{ 2020-03-11T22:53:59.425 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_log_t": { 2020-03-11T22:53:59.425 INFO:tasks.workunit.client.0.smithi045.stdout: "head": "34'150", 2020-03-11T22:53:59.425 INFO:tasks.workunit.client.0.smithi045.stdout: "tail": "34'148", 2020-03-11T22:53:59.425 INFO:tasks.workunit.client.0.smithi045.stdout: "log": [ 2020-03-11T22:53:59.425 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "modify", 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'149", 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "prior_version": "0'0", 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4633.0:1", 2020-03-11T22:53:59.426 INFO:tasks.workunit.client.0.smithi045.stdout: "extra_reqids": [], 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "mtime": "2020-03-11T22:53:21.397213+0000", 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": 0, 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "mod_desc": { 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "object_mod_desc": { 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "can_local_rollback": false, 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "rollback_info_completed": false, 2020-03-11T22:53:59.427 INFO:tasks.workunit.client.0.smithi045.stdout: "ops": [] 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_regions": { 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "object_clean_regions": { 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_omap": true, 2020-03-11T22:53:59.428 INFO:tasks.workunit.client.0.smithi045.stdout: "new_object": false 2020-03-11T22:53:59.429 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.429 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.429 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.429 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.429 INFO:tasks.workunit.client.0.smithi045.stdout: "op": "modify", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'150", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "prior_version": "0'0", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4636.0:1", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "extra_reqids": [], 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "mtime": "2020-03-11T22:53:21.547902+0000", 2020-03-11T22:53:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": 0, 2020-03-11T22:53:59.431 INFO:tasks.workunit.client.0.smithi045.stdout: "mod_desc": { 2020-03-11T22:53:59.431 INFO:tasks.workunit.client.0.smithi045.stdout: "object_mod_desc": { 2020-03-11T22:53:59.431 INFO:tasks.workunit.client.0.smithi045.stdout: "can_local_rollback": false, 2020-03-11T22:53:59.431 INFO:tasks.workunit.client.0.smithi045.stdout: "rollback_info_completed": false, 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: "ops": [] 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_regions": { 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: "object_clean_regions": { 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:59.432 INFO:tasks.workunit.client.0.smithi045.stdout: "clean_omap": true, 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: "new_object": false 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: ], 2020-03-11T22:53:59.433 INFO:tasks.workunit.client.0.smithi045.stdout: "dups": [ 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4609.0:1", 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'141", 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "141", 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.434 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4612.0:1", 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'142", 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "142", 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.435 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4615.0:1", 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'143", 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "143", 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.436 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4618.0:1", 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'144", 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "144", 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.437 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4621.0:1", 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'145", 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "145", 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4624.0:1", 2020-03-11T22:53:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'146", 2020-03-11T22:53:59.439 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "146", 2020-03-11T22:53:59.439 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.439 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.439 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4627.0:1", 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'147", 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "147", 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.440 INFO:tasks.workunit.client.0.smithi045.stdout: { 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: "reqid": "client.4630.0:1", 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: "version": "34'148", 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: "user_version": "148", 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: "return_code": "0" 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.441 INFO:tasks.workunit.client.0.smithi045.stdout: ] 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout: }, 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout: "pg_missing_t": { 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout: "missing": [], 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout: "may_include_deletes": true 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout: } 2020-03-11T22:53:59.442 INFO:tasks.workunit.client.0.smithi045.stdout:} 2020-03-11T22:53:59.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:53:59.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:53:59.445 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to max 2 2020-03-11T22:53:59.445 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:59.445 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:59.446 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:59.446 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:59.446 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:59.446 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:59.447 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:59.447 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:59.447 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:59.447 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:59.448 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:59.448 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to max 2 2020-03-11T22:53:59.448 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:59.449 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:59.449 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:59.449 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:59.449 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:59.450 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:59.450 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:59.450 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:59.450 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:59.450 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:59.451 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:59.451 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to max 2 2020-03-11T22:53:59.451 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:59.452 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:59.452 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:59.452 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:59.452 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:59.453 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:59.455 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:59.455 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:59.456 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:59.456 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:59.456 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:24.356+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:59.456 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_after v 34'150 2020-03-11T22:53:59.457 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:59.457 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:59.457 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:59.458 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:59.458 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:59.458 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:59.458 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'149 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi045.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:53.790+0000 7fb5997db700 20 copy_up_to/copy_after copy dup from log version 34'150 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:59.462 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:59.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:59.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:59.462 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:59.464 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:59.464 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:59.465 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:59.465 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:59.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:59.466 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:59.466 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:59.470 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:59.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:59.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:59.471 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:59.471 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:59.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:59.472 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:59.487 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:59.488 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.488 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:53:59.488 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76792 2020-03-11T22:53:59.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:59.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:59.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:53:59.492 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:59.493 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:59.494 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:59.495 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:59.496 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:59.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:59.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:59.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:59.498 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:59.498 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.498 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:53:59.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76792 2020-03-11T22:53:59.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:59.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:59.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:53:59.500 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:53:59.501 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.501 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:53:59.501 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76792 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:53:59.504 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:53:59.505 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:53:59.506 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:53:59.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:53:59.507 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:53:59.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:59.853 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:59.854 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:59.854 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.854 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:53:59.854 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:53:59.854 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:54:00.068 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:54:00.071 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:54:00.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:00.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:00.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:54:00.072 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:54:00.073 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:00.073 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:00.073 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:00.073 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:00.074 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:00.074 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:00.074 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76792/ceph-mon.a.asok 2020-03-11T22:54:00.074 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:00.075 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76792/ceph-mon.a.asok config get fsid 2020-03-11T22:54:00.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:54:00.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:00.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:00.173 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:00.174 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:00.175 INFO:tasks.workunit.client.0.smithi045.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:00.175 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76792/ceph-mon.a.asok 2020-03-11T22:54:00.175 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:00.175 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76792/ceph-mon.a.asok config get mon_host 2020-03-11T22:54:00.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:54:00.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:00.265 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:54:00.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:54:00.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:54:00.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:54:00.266 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:01.163 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:01.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:01.164 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:01.164 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:01.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:54:01.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:01.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:01.189 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:54:01.198 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:54:01.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:01.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:54:01.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:01.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:01.199 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:54:01.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:01.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:54:01.200 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:01.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:01.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:01.201 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:01.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:54:01.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:54:01.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:01.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:01.203 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:01.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:01.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:01.203 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:01.204 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:01.204 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:01.204 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:01.204 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:01.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:01.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:01.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:01.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:01.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:01.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:01.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:01.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:54:01.208 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:01.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5b88adce-c850-4162-83d2-0b3f53f5b54e 2020-03-11T22:54:01.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 5b88adce-c850-4162-83d2-0b3f53f5b54e' 2020-03-11T22:54:01.208 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:01.209 INFO:tasks.workunit.client.0.smithi045.stdout:add osd0 5b88adce-c850-4162-83d2-0b3f53f5b54e 2020-03-11T22:54:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAJbGlebwTGDBAAF0ynTbKe9R4U15c3Rk+1Xg== 2020-03-11T22:54:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAJbGlebwTGDBAAF0ynTbKe9R4U15c3Rk+1Xg=="}' 2020-03-11T22:54:01.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5b88adce-c850-4162-83d2-0b3f53f5b54e -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:01.584 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:01.594 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:01.597 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJbGlebwTGDBAAF0ynTbKe9R4U15c3Rk+1Xg== --osd-uuid 5b88adce-c850-4162-83d2-0b3f53f5b54e 2020-03-11T22:54:01.619 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.616+0000 7fa03c804ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.621 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.618+0000 7fa03c804ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.622 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.619+0000 7fa03c804ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.622 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.620+0000 7fa03c804ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.622 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.620+0000 7fa03c804ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.623 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.620+0000 7fa03c804ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.667 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:01.664+0000 7fa03c804ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:54:04.813 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:54:04.814 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:04.814 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd0 key to auth repository 2020-03-11T22:54:04.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:54:04.815 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:05.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:54:05.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:05.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:05.053 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:05.054 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:05.054 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.0 2020-03-11T22:54:05.074 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:05.072+0000 7f14e8af5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:05.102 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:05.099+0000 7f14e8af5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:05.102 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:05.100+0000 7f14e8af5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:05.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:54:05.427 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:05.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:54:05.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:05.428 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:05.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:05.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:05.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:05.429 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:05.430 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:05.617 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:05.614+0000 7f14e8af5ec0 -1 Falling back to public interface 2020-03-11T22:54:05.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:06.408 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:06.405+0000 7f14e8af5ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:54:06.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:06.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:06.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:06.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:06.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:06.793 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:07.183 INFO:tasks.workunit.client.0.smithi045.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85303,v1:127.0.0.1:6803/85303] [v2:127.0.0.1:6804/85303,v1:127.0.0.1:6805/85303] exists,up 5b88adce-c850-4162-83d2-0b3f53f5b54e 2020-03-11T22:54:07.183 INFO:tasks.workunit.client.0.smithi045.stdout:add osd1 2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:54:07.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:07.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:07.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:07.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:07.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:54:07.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:54:07.186 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:07.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:07.187 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:07.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:07.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:07.187 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:07.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:07.189 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8 2020-03-11T22:54:07.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8' 2020-03-11T22:54:07.191 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:07.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAPbGleovCJCxAAxGFOAxOU5RPP+xB9JMeXWA== 2020-03-11T22:54:07.196 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAPbGleovCJCxAAxGFOAxOU5RPP+xB9JMeXWA=="}' 2020-03-11T22:54:07.197 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:07.660 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:07.670 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:07.671 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPbGleovCJCxAAxGFOAxOU5RPP+xB9JMeXWA== --osd-uuid 2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8 2020-03-11T22:54:07.696 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.693+0000 7f208b0a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.697 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.695+0000 7f208b0a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.699 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.696+0000 7f208b0a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.699 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.696+0000 7f208b0a7ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.699 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.696+0000 7f208b0a7ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.700 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.696+0000 7f208b0a7ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.764 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:07.761+0000 7f208b0a7ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:54:11.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:54:11.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:11.184 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:54:11.185 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:11.185 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd1 key to auth repository 2020-03-11T22:54:11.667 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:54:11.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:11.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:11.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:11.668 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:11.668 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.1 2020-03-11T22:54:11.690 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:11.687+0000 7fec2c610ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.691 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:11.689+0000 7fec2c610ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.693 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:11.690+0000 7fec2c610ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:12.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:12.033 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:12.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:12.034 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:12.034 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:12.208 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:12.205+0000 7fec2c610ec0 -1 Falling back to public interface 2020-03-11T22:54:12.397 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:13.039 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:13.035+0000 7fec2c610ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:54:13.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:13.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:13.398 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:13.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:13.399 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:13.399 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi045.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85658,v1:127.0.0.1:6811/85658] [v2:127.0.0.1:6812/85658,v1:127.0.0.1:6813/85658] exists,up 2e7fbd75-32b4-4969-bbd5-e01ec7a6c0d8 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi045.stdout:add osd2 bc913d02-07a6-47b9-89ae-9e6b34729885 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:13.795 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:13.796 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:13.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bc913d02-07a6-47b9-89ae-9e6b34729885 2020-03-11T22:54:13.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 bc913d02-07a6-47b9-89ae-9e6b34729885' 2020-03-11T22:54:13.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:13.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAVbGleS8feLxAA/qgftI/gFCQuz2FXx4quIA== 2020-03-11T22:54:13.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAVbGleS8feLxAA/qgftI/gFCQuz2FXx4quIA=="}' 2020-03-11T22:54:13.807 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bc913d02-07a6-47b9-89ae-9e6b34729885 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:14.435 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:54:14.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:14.446 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAVbGleS8feLxAA/qgftI/gFCQuz2FXx4quIA== --osd-uuid bc913d02-07a6-47b9-89ae-9e6b34729885 2020-03-11T22:54:14.470 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.466+0000 7f104f02eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.472 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.468+0000 7f104f02eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.473 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.469+0000 7f104f02eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.473 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.469+0000 7f104f02eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:14.474 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.469+0000 7f104f02eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:14.474 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.469+0000 7f104f02eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:14.512 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:14.509+0000 7f104f02eec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:54:18.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:54:18.156 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:18.157 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd2 key to auth repository 2020-03-11T22:54:18.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:54:18.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:18.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:54:18.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:18.643 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:18.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:18.644 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:18.644 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.2 2020-03-11T22:54:18.665 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:18.661+0000 7fbeabf06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.667 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:18.663+0000 7fbeabf06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.669 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:18.665+0000 7fbeabf06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:19.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:54:19.007 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:19.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:19.009 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:19.184 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:19.181+0000 7fbeabf06ec0 -1 Falling back to public interface 2020-03-11T22:54:19.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:19.954 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:19.950+0000 7fbeabf06ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:54:20.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:20.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:20.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:20.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:20.413 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:20.413 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/86013,v1:127.0.0.1:6819/86013] [v2:127.0.0.1:6820/86013,v1:127.0.0.1:6821/86013] exists,up bc913d02-07a6-47b9-89ae-9e6b34729885 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stdout:add osd3 6fde3cd8-de08-4e1e-b1e2-951bca592e42 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:20.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:54:20.867 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:20.868 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:20.869 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:20.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:20.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6fde3cd8-de08-4e1e-b1e2-951bca592e42 2020-03-11T22:54:20.872 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 6fde3cd8-de08-4e1e-b1e2-951bca592e42' 2020-03-11T22:54:20.872 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:20.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAcbGleTb5ANBAAYFPHX5oUseTTZH7d97z5Xw== 2020-03-11T22:54:20.879 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAcbGleTb5ANBAAYFPHX5oUseTTZH7d97z5Xw=="}' 2020-03-11T22:54:20.880 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6fde3cd8-de08-4e1e-b1e2-951bca592e42 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:21.391 INFO:tasks.workunit.client.0.smithi045.stdout:3 2020-03-11T22:54:21.401 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:21.402 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAcbGleTb5ANBAAYFPHX5oUseTTZH7d97z5Xw== --osd-uuid 6fde3cd8-de08-4e1e-b1e2-951bca592e42 2020-03-11T22:54:21.427 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.423+0000 7f3587681ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.428 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.425+0000 7f3587681ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.430 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.426+0000 7f3587681ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.431 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.427+0000 7f3587681ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:21.431 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.427+0000 7f3587681ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:21.431 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.427+0000 7f3587681ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:21.579 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:21.575+0000 7f3587681ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:54:24.756 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:54:24.756 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:24.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:54:24.757 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:24.757 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd3 key to auth repository 2020-03-11T22:54:25.242 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.3 2020-03-11T22:54:25.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:54:25.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:25.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:25.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:25.243 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:25.264 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:25.260+0000 7f9ff771dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.266 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:25.262+0000 7f9ff771dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.267 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:25.264+0000 7f9ff771dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:54:25.609 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:25.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:54:25.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:25.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:25.611 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:25.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:25.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:25.612 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:25.612 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:25.787 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:25.783+0000 7f9ff771dec0 -1 Falling back to public interface 2020-03-11T22:54:25.996 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:26.588 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:26.584+0000 7f9ff771dec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:26.998 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:27.438 INFO:tasks.workunit.client.0.smithi045.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86368,v1:127.0.0.1:6827/86368] [v2:127.0.0.1:6828/86368,v1:127.0.0.1:6829/86368] exists,up 6fde3cd8-de08-4e1e-b1e2-951bca592e42 2020-03-11T22:54:27.439 INFO:tasks.workunit.client.0.smithi045.stdout:add osd4 26adb011-c097-4816-a6ca-d600a839707c 2020-03-11T22:54:27.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:27.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:27.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:27.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:27.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:27.441 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:27.442 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:27.443 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:54:27.444 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:27.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=26adb011-c097-4816-a6ca-d600a839707c 2020-03-11T22:54:27.445 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 26adb011-c097-4816-a6ca-d600a839707c' 2020-03-11T22:54:27.445 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:27.451 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAjbGleUlC+GhAATGzhXWQuUBZLY4+d87zeKw== 2020-03-11T22:54:27.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAjbGleUlC+GhAATGzhXWQuUBZLY4+d87zeKw=="}' 2020-03-11T22:54:27.452 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 26adb011-c097-4816-a6ca-d600a839707c -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:27.992 INFO:tasks.workunit.client.0.smithi045.stdout:4 2020-03-11T22:54:28.002 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:28.003 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAjbGleUlC+GhAATGzhXWQuUBZLY4+d87zeKw== --osd-uuid 26adb011-c097-4816-a6ca-d600a839707c 2020-03-11T22:54:28.027 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.024+0000 7f1bc772aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:28.029 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.025+0000 7f1bc772aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:28.031 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.027+0000 7f1bc772aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:28.032 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.027+0000 7f1bc772aec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:28.032 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.027+0000 7f1bc772aec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:28.032 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.027+0000 7f1bc772aec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:28.071 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:28.067+0000 7f1bc772aec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:54:31.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:54:31.682 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:31.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:54:31.683 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:31.684 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd4 key to auth repository 2020-03-11T22:54:32.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:54:32.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:32.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:32.171 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.4 2020-03-11T22:54:32.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:32.172 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:32.193 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:32.190+0000 7fe76defdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:32.195 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:32.191+0000 7fe76defdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:32.197 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:32.193+0000 7fe76defdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:32.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:54:32.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:32.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:54:32.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:32.537 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:32.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:32.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:32.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:32.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:32.538 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:32.714 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:32.710+0000 7fe76defdec0 -1 Falling back to public interface 2020-03-11T22:54:32.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:33.439 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:33.435+0000 7fe76defdec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:54:33.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:33.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:33.938 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:33.939 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:33.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:33.939 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:34.339 INFO:tasks.workunit.client.0.smithi045.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/86723,v1:127.0.0.1:6835/86723] [v2:127.0.0.1:6836/86723,v1:127.0.0.1:6837/86723] exists,up 26adb011-c097-4816-a6ca-d600a839707c 2020-03-11T22:54:34.339 INFO:tasks.workunit.client.0.smithi045.stdout:add osd5 f36057e8-306b-44c9-9382-a988553d86f9 2020-03-11T22:54:34.339 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:34.342 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76792 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:34.343 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:34.344 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:34.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:54:34.345 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:34.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f36057e8-306b-44c9-9382-a988553d86f9 2020-03-11T22:54:34.345 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 f36057e8-306b-44c9-9382-a988553d86f9' 2020-03-11T22:54:34.345 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:34.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAqbGleHcniFBAAOYcG1O9FE3zs44/leCHarw== 2020-03-11T22:54:34.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAqbGleHcniFBAAOYcG1O9FE3zs44/leCHarw=="}' 2020-03-11T22:54:34.353 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f36057e8-306b-44c9-9382-a988553d86f9 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:34.953 INFO:tasks.workunit.client.0.smithi045.stdout:5 2020-03-11T22:54:34.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:34.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAqbGleHcniFBAAOYcG1O9FE3zs44/leCHarw== --osd-uuid f36057e8-306b-44c9-9382-a988553d86f9 2020-03-11T22:54:34.988 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.984+0000 7f28965fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.990 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.986+0000 7f28965fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.992 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.988+0000 7f28965fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.992 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.988+0000 7f28965fdec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:34.992 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.988+0000 7f28965fdec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:34.993 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:34.988+0000 7f28965fdec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:35.096 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:35.093+0000 7f28965fdec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:54:38.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:54:38.536 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:38.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:54:38.538 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:38.538 INFO:tasks.workunit.client.0.smithi045.stdout:adding osd5 key to auth repository 2020-03-11T22:54:39.027 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:54:39.027 INFO:tasks.workunit.client.0.smithi045.stdout:start osd.5 2020-03-11T22:54:39.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:39.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=2b661212-388b-4db5-ae16-4d56ab25329d --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76792/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:39.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:39.028 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:39.050 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:39.046+0000 7f708ce92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:39.051 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:39.048+0000 7f708ce92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:39.053 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:39.049+0000 7f708ce92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:39.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:54:39.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:39.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:54:39.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:39.415 INFO:tasks.workunit.client.0.smithi045.stdout:0 2020-03-11T22:54:39.570 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:39.566+0000 7f708ce92ec0 -1 Falling back to public interface 2020-03-11T22:54:39.778 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:40.399 INFO:tasks.workunit.client.0.smithi045.stderr:2020-03-11T22:54:40.395+0000 7f708ce92ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:54:40.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:40.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:40.779 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:40.779 INFO:tasks.workunit.client.0.smithi045.stdout:1 2020-03-11T22:54:40.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:40.780 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:41.157 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:42.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi045.stdout:2 2020-03-11T22:54:42.549 INFO:tasks.workunit.client.0.smithi045.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/87078,v1:127.0.0.1:6843/87078] [v2:127.0.0.1:6844/87078,v1:127.0.0.1:6845/87078] exists,up f36057e8-306b-44c9-9382-a988553d86f9 2020-03-11T22:54:42.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:42.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:42.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:42.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:54:42.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:54:43.143 INFO:tasks.workunit.client.0.smithi045.stderr:pool 'test' already exists 2020-03-11T22:54:43.151 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:54:44.152 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:54:44.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.153 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:54:44.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.790 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:54:44.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:45.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.064 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:45.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.214 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:45.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:45.293 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.294 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:45.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:45.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.623 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:45.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:45.772 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.773 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:45.856 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.857 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:45.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:45.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.964 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:46.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:46.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:46.224 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.225 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:46.262 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:46.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:46.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.406 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:46.439 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.440 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:46.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.517 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:46.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.554 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:46.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.676 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:46.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.715 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:46.817 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.818 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:46.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.894 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:46.929 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.930 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:47.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.010 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:47.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.107 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:47.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.202 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:47.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:47.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.350 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:47.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.396 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:47.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:47.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.503 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:47.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:47.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:47.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.707 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:47.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.741 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:54:47.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.890 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:54:47.962 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.963 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:54:47.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.999 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:54:48.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:54:48.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.149 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:54:48.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.188 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:54:48.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.263 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:54:48.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.338 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:54:48.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.414 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:54:48.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.449 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:54:48.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.648 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:54:48.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:54:48.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.722 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:54:48.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:54:48.912 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.912 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:54:48.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:54:49.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.023 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:54:49.182 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.183 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:54:49.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:54:49.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.256 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:54:49.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:54:49.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.422 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:54:49.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:54:49.533 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.533 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:54:49.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.642 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:54:49.716 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.717 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:54:49.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:54:49.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.871 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:54:49.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.906 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:54:49.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.984 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:54:50.160 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.161 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:54:50.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.198 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:54:50.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.233 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:54:50.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.310 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:54:50.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:54:50.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.408 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:54:50.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.527 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:54:50.686 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.687 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:54:50.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.725 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:54:50.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.758 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:54:50.912 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.912 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:54:50.970 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:54:51.008 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.009 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:54:51.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:54:51.204 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:54:51.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.242 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:54:51.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.298 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:54:51.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.405 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:54:51.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.442 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:54:51.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.484 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:54:51.631 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.631 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:54:51.667 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.667 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:54:51.739 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.740 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:54:51.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.776 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:54:51.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.898 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:54:51.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.934 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:54:51.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.967 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:54:52.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.040 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:54:52.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.104 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:54:52.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.216 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:54:52.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.255 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:54:52.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.375 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:54:52.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.458 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:54:52.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.497 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:54:52.625 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.626 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:54:52.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.664 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:54:52.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.742 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:54:52.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.859 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:54:52.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.897 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:54:52.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.977 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:54:53.051 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.052 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:54:53.168 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.169 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:54:53.207 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.208 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:54:53.335 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.336 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:54:53.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.374 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:54:53.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:54:53.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.486 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:54:53.550 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.551 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:54:53.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.628 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:54:53.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:54:53.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:54:53.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.827 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:54:53.869 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.870 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:54:53.954 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:54:54.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.032 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:54:54.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.121 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:54:54.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.275 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:54:54.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:54:54.358 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:54:54.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.430 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:54:54.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.469 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:54:54.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.622 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:54:54.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.662 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:54:54.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:54:54.850 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.851 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:54:54.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.886 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:54:54.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.995 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:54:55.044 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.045 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:54:55.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.115 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:54:55.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.190 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:54:55.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.291 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:54:55.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.328 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:54:55.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.361 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:54:55.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.473 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:54:55.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.520 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:54:55.616 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:54:55.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:55.661 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:54:55.809 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:55.809 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:56.155 INFO:tasks.workunit.client.0.smithi045.stderr:dumped pgs 2020-03-11T22:54:56.162 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:56.687 INFO:tasks.workunit.client.0.smithi045.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:56.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:56.695 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:57.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:57.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:57.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:57.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:57.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:57.698 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:57.698 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:57.699 INFO:tasks.workunit.client.0.smithi045.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:57.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:57.699 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:57.700 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:57.805 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:57.806 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:57.806 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:58.169 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:1 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:2 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:3 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:4 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:5' 2020-03-11T22:54:58.170 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:58.171 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.171 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:58.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:54:58.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:54:58.300 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.301 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:58.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:54:58.431 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2020-03-11T22:54:58.432 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.432 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:58.561 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-11T22:54:58.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542154' 2020-03-11T22:54:58.562 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.562 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:58.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378633 2020-03-11T22:54:58.691 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542154 3-81604378633' 2020-03-11T22:54:58.692 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.692 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:58.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215111 2020-03-11T22:54:58.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542154 3-81604378633 4-103079215111' 2020-03-11T22:54:58.824 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:58.824 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051591 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542154 3-81604378633 4-103079215111 5-124554051591' 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.956 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:58.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:58.956 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:54:58.957 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:54:58.957 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:58.957 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:54:59.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:54:59.321 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:59.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:54:59.322 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:59.323 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:59.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:54:59.323 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:59.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:54:59.324 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:54:59.324 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:59.325 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:54:59.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-11T22:54:59.801 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:59.801 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-11T22:54:59.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:59.802 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:59.802 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-11T22:54:59.803 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:59.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-11T22:54:59.804 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-11T22:54:59.804 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:59.804 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.2 seq 60129542154 2020-03-11T22:55:00.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542154 -lt 60129542154 2020-03-11T22:55:00.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:55:00.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378633 2020-03-11T22:55:00.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:55:00.220 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:55:00.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378633 2020-03-11T22:55:00.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:55:00.221 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378633 2020-03-11T22:55:00.222 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2020-03-11T22:55:00.222 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:55:00.222 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.3 seq 81604378633 2020-03-11T22:55:00.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378633 -lt 81604378633 2020-03-11T22:55:00.588 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:55:00.588 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215111 2020-03-11T22:55:00.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:55:00.589 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:55:00.589 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215111 2020-03-11T22:55:00.590 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:55:00.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215111 2020-03-11T22:55:00.590 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215111' 2020-03-11T22:55:00.591 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:55:00.591 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.4 seq 103079215111 2020-03-11T22:55:01.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215109 -lt 103079215111 2020-03-11T22:55:01.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:55:02.138 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:55:02.139 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:55:02.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215112 -lt 103079215111 2020-03-11T22:55:02.518 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:55:02.519 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-124554051591 2020-03-11T22:55:02.519 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:55:02.519 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:55:02.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-124554051591 2020-03-11T22:55:02.520 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:55:02.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051591 2020-03-11T22:55:02.521 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 124554051591' 2020-03-11T22:55:02.521 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:55:02.521 INFO:tasks.workunit.client.0.smithi045.stdout:waiting osd.5 seq 124554051591 2020-03-11T22:55:02.883 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051591 -lt 124554051591 2020-03-11T22:55:02.883 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:55:02.884 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:02.884 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:03.377 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:55:03.379 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:03.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:03.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:03.379 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:03.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:03.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:03.380 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:03.732 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:03.732 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:03.732 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:03.733 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:04.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:04.227 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:55:04.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:55:04.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:55:04.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:04.228 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:04.329 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:04.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:04.330 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:04.747 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:04.747 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:04.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:04.748 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:05.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:05.218 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:05.219 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:05.220 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:05.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20213 2020-03-11T22:55:05.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20213 '!=' null 2020-03-11T22:55:05.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:05.697 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:05.698 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:05.797 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:05.798 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:05.798 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:05.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:05.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:05.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:05.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:05.799 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:06.150 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:06.150 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:06.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:06.151 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:06.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:06.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:06.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:06.640 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:06.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:06.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:06.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:06.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:06.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:07.113 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16545 2020-03-11T22:55:07.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16545 '!=' null 2020-03-11T22:55:07.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:07.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:07.114 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:07.215 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:07.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:07.216 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:07.563 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:07.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:07.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:07.564 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:08.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:08.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:08.056 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:08.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16545 2020-03-11T22:55:08.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16545 '!=' null 2020-03-11T22:55:08.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:08.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:08.565 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:08.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:08.666 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:08.666 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:09.024 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:09.024 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:09.024 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:09.024 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:09.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:09.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:09.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:09.498 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:09.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:09.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:09.499 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:09.499 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:09.499 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:09.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30511 2020-03-11T22:55:09.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30511 '!=' null 2020-03-11T22:55:09.982 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:09.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:09.983 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:10.083 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:10.083 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:10.084 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:10.435 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:10.435 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:10.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:10.436 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:10.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:10.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:10.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:10.908 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:10.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:10.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:10.909 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:10.909 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:10.909 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:11.382 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28749 2020-03-11T22:55:11.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28749 '!=' null 2020-03-11T22:55:11.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:11.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:11.383 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:11.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:11.483 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:11.484 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:11.838 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:11.839 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:11.839 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:11.839 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:12.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:12.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:12.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:12.315 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:12.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:12.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:12.316 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:12.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:12.316 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:12.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28749 2020-03-11T22:55:12.845 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28749 '!=' null 2020-03-11T22:55:12.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:12.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:12.846 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:12.946 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:12.947 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:12.948 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:13.302 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:13.302 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:13.302 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:13.302 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:13.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:13.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:13.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:13.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:13.781 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:13.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:13.782 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:13.782 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:13.782 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:14.257 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28749 2020-03-11T22:55:14.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28749 '!=' null 2020-03-11T22:55:14.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:14.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:14.258 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:14.359 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:14.360 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:14.360 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:14.710 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:14.710 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:14.711 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:14.711 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:15.205 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:15.206 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:15.206 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:15.206 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44848 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44848 '!=' null 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:15.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:15.775 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:15.775 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:15.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:15.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:15.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:15.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:15.776 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:16.130 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:16.130 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:16.130 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:16.130 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:16.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:16.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:16.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:16.606 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:16.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:16.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:16.607 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:16.607 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:16.607 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:17.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31623 2020-03-11T22:55:17.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31623 '!=' null 2020-03-11T22:55:17.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:17.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:17.122 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:17.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:17.223 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:17.223 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:17.223 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:17.223 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:17.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:17.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:17.224 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:17.573 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:17.604 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:17.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:17.605 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:18.054 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:18.055 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:18.056 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:18.056 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:18.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31623 2020-03-11T22:55:18.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31623 '!=' null 2020-03-11T22:55:18.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:18.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:18.540 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:18.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:18.641 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:18.641 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:18.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:18.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:18.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:18.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:18.642 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:18.992 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:18.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:18.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:18.993 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:19.477 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:19.478 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=47722 2020-03-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 47722 '!=' null 2020-03-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:20.059 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:20.060 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:20.060 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:20.412 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:20.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:20.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:20.413 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:20.936 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:20.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:20.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:20.937 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:20.937 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:20.937 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:21.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32198 2020-03-11T22:55:21.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32198 '!=' null 2020-03-11T22:55:21.410 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:21.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:21.411 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:21.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:21.511 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:21.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:21.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:21.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:21.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:21.512 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:21.513 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:21.866 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:21.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:21.866 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:21.867 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:22.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:22.340 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:22.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:22.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:22.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32198 2020-03-11T22:55:22.839 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32198 '!=' null 2020-03-11T22:55:22.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:22.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:22.840 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:22.941 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:22.942 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:22.942 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:23.290 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:23.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:23.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:23.291 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:23.767 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:23.768 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:23.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:23.769 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:24.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32198 2020-03-11T22:55:24.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32198 '!=' null 2020-03-11T22:55:24.239 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:24.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:24.240 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:24.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:24.341 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:24.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:24.341 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:24.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:24.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:24.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:24.342 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:24.693 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:24.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:24.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:24.694 INFO:tasks.workunit.client.0.smithi045.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:24.977 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:24.978 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2020-03-11T22:55:24.978 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2020-03-11T22:55:24.978 DEBUG:teuthology.exit:Finished running handlers